[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980676253068-24012-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:37:56 dist-test-slave-d645 krb5kdc[24019](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:37:56 dist-test-slave-d645 krb5kdc[24019](info): set up 2 sockets
Jun 20 18:37:56 dist-test-slave-d645 krb5kdc[24019](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260620 18:37:57.299103 24012 mini_kdc.cc:121] Time spent starting KDC: real 0.964s user 0.000s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980676253068-24012-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980676253068-24012-0/krb5kdc/test-user.keytab.
Jun 20 18:37:57 dist-test-slave-d645 krb5kdc[24019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980677, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:37:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:37:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.115.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.115.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.115.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab.
Entry for principal kudu/127.23.115.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab.
I20260620 18:37:57.464435 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980676253068-24012-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980676253068-24012-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980676253068-24012-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.115.62:32937
--webserver_interface=127.23.115.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab
--principal=kudu/127.23.115.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:37731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.115.62:32937
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:37:57.776371 24032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:37:57.776643 24032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:37:57.776759 24032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:37:57.783977 24032 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:37:57.784076 24032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:37:57.784143 24032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:37:57.784216 24032 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:37:57.784279 24032 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:37:57.794184 24032 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:37731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980676253068-24012-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980676253068-24012-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.115.62:32937
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab
--principal=kudu/127.23.115.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.115.62:32937
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.115.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980676253068-24012-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:37:57.795493 24032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:37:57.796784 24032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:37:57.808813 24040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:37:57.808859 24038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:37:57.809238 24032 server_base.cc:1034] running on GCE node
W20260620 18:37:57.809351 24037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:37:57.810277 24032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:37:57.811721 24032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:37:57.812924 24032 hybrid_clock.cc:648] HybridClock initialized: now 1781980677812874 us; error 36 us; skew 500 ppm
I20260620 18:37:57.813251 24032 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:37:57 dist-test-slave-d645 krb5kdc[24019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980677, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:37:57.816504 24032 init.cc:377] Logged in from keytab as kudu/127.23.115.62@KRBTEST.COM (short username kudu)
I20260620 18:37:57.817998 24032 webserver.cc:460] Webserver started at http://127.23.115.62:43425/ using document root <none> and password file <none>
I20260620 18:37:57.818656 24032 fs_manager.cc:359] Metadata directory not provided
I20260620 18:37:57.818765 24032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:37:57.819096 24032 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:37:57.821921 24032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980676253068-24012-0/minicluster-data/master-0/data/instance:
uuid: "b42f411ff29f4233ba18021a6410919f"
format_stamp: "Formatted at 2026-06-20 18:37:57 on dist-test-slave-d645"
server_key: "babf54003121eab2121caf9f3db99661"
server_key_iv: "95133d17b6eb4002d35ad152c04f380c"
server_key_version: "encryptionkey@0"
I20260620 18:37:57.822672 24032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980676253068-24012-0/minicluster-data/master-0/wal/instance:
uuid: "b42f411ff29f4233ba18021a6410919f"
format_stamp: "Formatted at 2026-06-20 18:37:57 on dist-test-slave-d645"
server_key: "babf54003121eab2121caf9f3db99661"
server_key_iv: "95133d17b6eb4002d35ad152c04f380c"
server_key_version: "encryptionkey@0"
I20260620 18:37:57.827932 24032 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 18:37:57.831705 24047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:37:57.833348 24032 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:37:57.833557 24032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980676253068-24012-0/minicluster-data/master-0/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980676253068-24012-0/minicluster-data/master-0/wal
uuid: "b42f411ff29f4233ba18021a6410919f"
format_stamp: "Formatted at 2026-06-20 18:37:57 on dist-test-slave-d645"
server_key: "babf54003121eab2121caf9f3db99661"
server_key_iv: "95133d17b6eb4002d35ad152c04f380c"
server_key_version: "encryptionkey@0"
I20260620 18:37:57.833765 24032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980676253068-24012-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980676253068-24012-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980676253068-24012-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:37:57.849846 24032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:37:57.854288 24032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:37:57.854569 24032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:37:57.874075 24032 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.62:32937
I20260620 18:37:57.874094 24098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.62:32937 every 8 connection(s)
I20260620 18:37:57.875455 24032 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
I20260620 18:37:57.879235 24099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:37:57.879721 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 24032
I20260620 18:37:57.880049 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980676253068-24012-0/minicluster-data/master-0/wal/instance
I20260620 18:37:57.882468 24012 external_mini_cluster.cc:1428] Setting key 90957e2a1b0bc098383685b51793bc4b
I20260620 18:37:57.891244 24099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b42f411ff29f4233ba18021a6410919f: Bootstrap starting.
I20260620 18:37:57.894057 24099 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b42f411ff29f4233ba18021a6410919f: Neither blocks nor log segments found. Creating new log.
I20260620 18:37:57.895051 24099 log.cc:824] T 00000000000000000000000000000000 P b42f411ff29f4233ba18021a6410919f: Log is configured to *not* fsync() on all Append() calls
I20260620 18:37:57.898294 24099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b42f411ff29f4233ba18021a6410919f: No bootstrap required, opened a new log
I20260620 18:37:57.905992 24099 raft_consensus.cc:348] T 00000000000000000000000000000000 P b42f411ff29f4233ba18021a6410919f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b42f411ff29f4233ba18021a6410919f" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 32937 } }
I20260620 18:37:57.906368 24099 raft_consensus.cc:374] T 00000000000000000000000000000000 P b42f411ff29f4233ba18021a6410919f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:37:57.906461 24099 raft_consensus.cc:729] T 00000000000000000000000000000000 P b42f411ff29f4233ba18021a6410919f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b42f411ff29f4233ba18021a6410919f, State: Initialized, Role: FOLLOWER
I20260620 18:37:57.907142 24099 consensus_queue.cc:260] T 00000000000000000000000000000000 P b42f411ff29f4233ba18021a6410919f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b42f411ff29f4233ba18021a6410919f" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 32937 } }
I20260620 18:37:57.907361 24099 raft_consensus.cc:388] T 00000000000000000000000000000000 P b42f411ff29f4233ba18021a6410919f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:37:57.907486 24099 raft_consensus.cc:482] T 00000000000000000000000000000000 P b42f411ff29f4233ba18021a6410919f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:37:57.907657 24099 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b42f411ff29f4233ba18021a6410919f [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:37:57.910789 24099 raft_consensus.cc:504] T 00000000000000000000000000000000 P b42f411ff29f4233ba18021a6410919f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b42f411ff29f4233ba18021a6410919f" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 32937 } }
Jun 20 18:37:57 dist-test-slave-d645 krb5kdc[24019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980677, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.115.62@KRBTEST.COM
I20260620 18:37:57.911347 24099 leader_election.cc:302] T 00000000000000000000000000000000 P b42f411ff29f4233ba18021a6410919f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b42f411ff29f4233ba18021a6410919f; no voters:
I20260620 18:37:57.911929 24099 leader_election.cc:288] T 00000000000000000000000000000000 P b42f411ff29f4233ba18021a6410919f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:37:57.912051 24104 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b42f411ff29f4233ba18021a6410919f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:37:57.913211 24104 raft_consensus.cc:686] T 00000000000000000000000000000000 P b42f411ff29f4233ba18021a6410919f [term 1 LEADER]: Becoming Leader. State: Replica: b42f411ff29f4233ba18021a6410919f, State: Running, Role: LEADER
I20260620 18:37:57.913775 24104 consensus_queue.cc:237] T 00000000000000000000000000000000 P b42f411ff29f4233ba18021a6410919f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b42f411ff29f4233ba18021a6410919f" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 32937 } }
I20260620 18:37:57.914304 24099 sys_catalog.cc:564] T 00000000000000000000000000000000 P b42f411ff29f4233ba18021a6410919f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:37:57.916711 24106 sys_catalog.cc:455] T 00000000000000000000000000000000 P b42f411ff29f4233ba18021a6410919f [sys.catalog]: SysCatalogTable state changed. Reason: New leader b42f411ff29f4233ba18021a6410919f. Latest consensus state: current_term: 1 leader_uuid: "b42f411ff29f4233ba18021a6410919f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b42f411ff29f4233ba18021a6410919f" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 32937 } } }
I20260620 18:37:57.916813 24105 sys_catalog.cc:455] T 00000000000000000000000000000000 P b42f411ff29f4233ba18021a6410919f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b42f411ff29f4233ba18021a6410919f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b42f411ff29f4233ba18021a6410919f" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 32937 } } }
I20260620 18:37:57.917196 24106 sys_catalog.cc:458] T 00000000000000000000000000000000 P b42f411ff29f4233ba18021a6410919f [sys.catalog]: This master's current role is: LEADER
I20260620 18:37:57.917202 24105 sys_catalog.cc:458] T 00000000000000000000000000000000 P b42f411ff29f4233ba18021a6410919f [sys.catalog]: This master's current role is: LEADER
I20260620 18:37:57.918058 24109 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:37:57.924744 24101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:57.886906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39678 (local address 127.23.115.62:32937)
0620 18:37:57.887803 (+ 897us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:57.887818 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:57.889011 (+ 1193us) server_negotiation.cc:407] Connection header received
0620 18:37:57.897909 (+ 8898us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:57.897935 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:57.898392 (+ 457us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:57.898810 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:57.902179 (+ 3369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:57.903662 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:57.905262 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:57.905893 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:57.906075 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:57.912696 (+ 6621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:57.912708 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:57.912727 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:37:57.912772 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:57.916685 (+ 3913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:57.917799 (+ 1114us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:57.917808 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:57.917815 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:57.918037 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:57.918612 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:57.918617 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:57.918620 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:57.919316 (+ 696us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:57.919590 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:57.922614 (+ 3024us) server_negotiation.cc:299] Negotiation successful
0620 18:37:57.923083 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":615,"thread_start_us":346,"threads_started":1}
I20260620 18:37:57.930291 24109 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:37:57.946075 24109 catalog_manager.cc:1347] Generated new cluster ID: 158366cc79524206a5a250bcca7a6eb8
I20260620 18:37:57.946219 24109 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:37:57.966316 24109 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:37:57.968076 24109 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:37:57.977404 24109 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b42f411ff29f4233ba18021a6410919f: Generated new TSK 0
I20260620 18:37:57.978250 24109 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.115.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.115.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.115.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.1.keytab.
Entry for principal kudu/127.23.115.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.1.keytab.
I20260620 18:37:58.038391 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980676253068-24012-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980676253068-24012-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980676253068-24012-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980676253068-24012-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.115.1:0
--local_ip_for_outbound_sockets=127.23.115.1
--webserver_interface=127.23.115.1
--webserver_port=0
--tserver_master_addrs=127.23.115.62:32937
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.1.keytab
--principal=kudu/127.23.115.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:37731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:37:58.382520 24125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:37:58.382818 24125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:37:58.382988 24125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:37:58.390491 24125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:37:58.390611 24125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:37:58.390779 24125 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.115.1
I20260620 18:37:58.401051 24125 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:37731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980676253068-24012-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980676253068-24012-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.1.keytab
--principal=kudu/127.23.115.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.115.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980676253068-24012-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.115.1
--webserver_port=0
--tserver_master_addrs=127.23.115.62:32937
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.115.1
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980676253068-24012-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:37:58.402288 24125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:37:58.403626 24125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:37:58.416077 24131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:37:58.416122 24130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:37:58.416567 24133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:37:58.417030 24125 server_base.cc:1034] running on GCE node
I20260620 18:37:58.417562 24125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:37:58.418443 24125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:37:58.419641 24125 hybrid_clock.cc:648] HybridClock initialized: now 1781980678419570 us; error 64 us; skew 500 ppm
I20260620 18:37:58.419937 24125 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:37:58 dist-test-slave-d645 krb5kdc[24019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980678, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:37:58.423038 24125 init.cc:377] Logged in from keytab as kudu/127.23.115.1@KRBTEST.COM (short username kudu)
I20260620 18:37:58.424408 24125 webserver.cc:460] Webserver started at http://127.23.115.1:45921/ using document root <none> and password file <none>
I20260620 18:37:58.425038 24125 fs_manager.cc:359] Metadata directory not provided
I20260620 18:37:58.425136 24125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:37:58.425470 24125 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:37:58.428411 24125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980676253068-24012-0/minicluster-data/ts-0/data/instance:
uuid: "1f006b294379400b8e7e6d29a4321011"
format_stamp: "Formatted at 2026-06-20 18:37:58 on dist-test-slave-d645"
server_key: "e6e76da5348a3d25ee0302ce5b59eedb"
server_key_iv: "f90dc76f47e39462b68ef8b6c8e4c9e5"
server_key_version: "encryptionkey@0"
I20260620 18:37:58.429144 24125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980676253068-24012-0/minicluster-data/ts-0/wal/instance:
uuid: "1f006b294379400b8e7e6d29a4321011"
format_stamp: "Formatted at 2026-06-20 18:37:58 on dist-test-slave-d645"
server_key: "e6e76da5348a3d25ee0302ce5b59eedb"
server_key_iv: "f90dc76f47e39462b68ef8b6c8e4c9e5"
server_key_version: "encryptionkey@0"
I20260620 18:37:58.433904 24125 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 18:37:58.437331 24140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:37:58.438651 24125 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 18:37:58.438791 24125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980676253068-24012-0/minicluster-data/ts-0/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980676253068-24012-0/minicluster-data/ts-0/wal
uuid: "1f006b294379400b8e7e6d29a4321011"
format_stamp: "Formatted at 2026-06-20 18:37:58 on dist-test-slave-d645"
server_key: "e6e76da5348a3d25ee0302ce5b59eedb"
server_key_iv: "f90dc76f47e39462b68ef8b6c8e4c9e5"
server_key_version: "encryptionkey@0"
I20260620 18:37:58.439013 24125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980676253068-24012-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980676253068-24012-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980676253068-24012-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:37:58.458617 24125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:37:58.462783 24125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:37:58.463093 24125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:37:58.464052 24125 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:37:58.465376 24125 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:37:58.465451 24125 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:37:58.465576 24125 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:37:58.465631 24125 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:37:58.495471 24125 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.1:36651
I20260620 18:37:58.495489 24252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.1:36651 every 8 connection(s)
I20260620 18:37:58.496861 24125 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980676253068-24012-0/minicluster-data/ts-0/data/info.pb
I20260620 18:37:58.506222 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 24125
I20260620 18:37:58.506443 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980676253068-24012-0/minicluster-data/ts-0/wal/instance
I20260620 18:37:58.507052 24012 external_mini_cluster.cc:1428] Setting key cccd478f1ea0170fc42928e47173c4f1
Jun 20 18:37:58 dist-test-slave-d645 krb5kdc[24019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980678, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.1@KRBTEST.COM for kudu/127.23.115.62@KRBTEST.COM
I20260620 18:37:58.516703 24255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:58.499369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.1:34913 (local address 127.23.115.62:32937)
0620 18:37:58.499874 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:58.499879 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:58.500845 (+ 966us) server_negotiation.cc:407] Connection header received
0620 18:37:58.502649 (+ 1804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:58.502656 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:58.502737 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:58.502834 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:58.504448 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:58.505158 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:58.506695 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:58.507132 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:58.507261 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:58.510899 (+ 3638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:58.510904 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:58.510907 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:37:58.510961 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:58.513623 (+ 2662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:58.514391 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:58.514396 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:58.514400 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:58.514522 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:58.514962 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:58.514967 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:58.514971 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:58.515289 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:58.515419 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:58.516121 (+ 702us) server_negotiation.cc:299] Negotiation successful
0620 18:37:58.516347 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":292,"threads_started":1}
I20260620 18:37:58.517634 24256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:58.499940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:32937 (local address 127.23.115.1:34913)
0620 18:37:58.500672 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:58.500721 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 18:37:58.502391 (+ 1670us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:37:58.503097 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:37:58.503175 (+ 78us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:37:58.503699 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:37:58.504247 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:58.504264 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:58.505316 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:58.505320 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:58.506565 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:58.506577 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:58.507357 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:58.507364 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:58.507606 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:58.508322 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:37:58.508359 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:37:58.510694 (+ 2335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:37:58.513768 (+ 3074us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:37:58.513777 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:37:58.513794 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:37:58.514278 (+ 484us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:37:58.514651 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:37:58.514656 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:37:58.514659 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:37:58.514838 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:37:58.515408 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:37:58.515416 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:37:58.515759 (+ 343us) client_negotiation.cc:769] Sending connection context
0620 18:37:58.516002 (+ 243us) client_negotiation.cc:240] Negotiation successful
0620 18:37:58.516511 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":283,"threads_started":1}
I20260620 18:37:58.519256 24253 heartbeater.cc:347] Connected to a master server at 127.23.115.62:32937
I20260620 18:37:58.519613 24253 heartbeater.cc:464] Registering TS with master...
I20260620 18:37:58.520481 24253 heartbeater.cc:511] Master 127.23.115.62:32937 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.23.115.2@KRBTEST.COM; defaulting to no policy
I20260620 18:37:58.522707 24064 ts_manager.cc:194] Registered new tserver with Master: 1f006b294379400b8e7e6d29a4321011 (127.23.115.1:36651)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.115.2@KRBTEST.COM" created.
I20260620 18:37:58.524305 24064 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.115.1@KRBTEST.COM'} at 127.23.115.1:34913
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.115.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.2.keytab.
Entry for principal kudu/127.23.115.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.2.keytab.
I20260620 18:37:58.542303 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980676253068-24012-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980676253068-24012-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980676253068-24012-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980676253068-24012-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.115.2:0
--local_ip_for_outbound_sockets=127.23.115.2
--webserver_interface=127.23.115.2
--webserver_port=0
--tserver_master_addrs=127.23.115.62:32937
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.2.keytab
--principal=kudu/127.23.115.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:37731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:37:58.858981 24259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:37:58.859266 24259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:37:58.859406 24259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:37:58.866798 24259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:37:58.866951 24259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:37:58.867122 24259 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.115.2
I20260620 18:37:58.876818 24259 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:37731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980676253068-24012-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980676253068-24012-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.2.keytab
--principal=kudu/127.23.115.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.115.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980676253068-24012-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.115.2
--webserver_port=0
--tserver_master_addrs=127.23.115.62:32937
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.115.2
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980676253068-24012-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:37:58.878049 24259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:37:58.879320 24259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:37:58.892441 24265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:37:58.892454 24264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:37:58.892478 24267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:37:58.893839 24259 server_base.cc:1034] running on GCE node
I20260620 18:37:58.894384 24259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:37:58.895481 24259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:37:58.896706 24259 hybrid_clock.cc:648] HybridClock initialized: now 1781980678896655 us; error 38 us; skew 500 ppm
I20260620 18:37:58.897023 24259 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:37:58 dist-test-slave-d645 krb5kdc[24019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980678, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:37:58.900202 24259 init.cc:377] Logged in from keytab as kudu/127.23.115.2@KRBTEST.COM (short username kudu)
I20260620 18:37:58.901638 24259 webserver.cc:460] Webserver started at http://127.23.115.2:37457/ using document root <none> and password file <none>
I20260620 18:37:58.902283 24259 fs_manager.cc:359] Metadata directory not provided
I20260620 18:37:58.902387 24259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:37:58.902690 24259 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:37:58.905535 24259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980676253068-24012-0/minicluster-data/ts-1/data/instance:
uuid: "4579bdedd0ae42d2b4585c694727e714"
format_stamp: "Formatted at 2026-06-20 18:37:58 on dist-test-slave-d645"
server_key: "216794c96dca80e2b59fa03efdbbf57c"
server_key_iv: "8a552b0b5c0d7bb46ab125f25f421667"
server_key_version: "encryptionkey@0"
I20260620 18:37:58.906266 24259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980676253068-24012-0/minicluster-data/ts-1/wal/instance:
uuid: "4579bdedd0ae42d2b4585c694727e714"
format_stamp: "Formatted at 2026-06-20 18:37:58 on dist-test-slave-d645"
server_key: "216794c96dca80e2b59fa03efdbbf57c"
server_key_iv: "8a552b0b5c0d7bb46ab125f25f421667"
server_key_version: "encryptionkey@0"
I20260620 18:37:58.911449 24259 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 18:37:58.915256 24274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:37:58.916596 24259 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:37:58.916749 24259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980676253068-24012-0/minicluster-data/ts-1/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980676253068-24012-0/minicluster-data/ts-1/wal
uuid: "4579bdedd0ae42d2b4585c694727e714"
format_stamp: "Formatted at 2026-06-20 18:37:58 on dist-test-slave-d645"
server_key: "216794c96dca80e2b59fa03efdbbf57c"
server_key_iv: "8a552b0b5c0d7bb46ab125f25f421667"
server_key_version: "encryptionkey@0"
I20260620 18:37:58.916934 24259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980676253068-24012-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980676253068-24012-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980676253068-24012-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:37:58.928354 24259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:37:58.932685 24259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:37:58.932971 24259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:37:58.933909 24259 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:37:58.935322 24259 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:37:58.935408 24259 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:37:58.935521 24259 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:37:58.935581 24259 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:37:58.965960 24259 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.2:39483
I20260620 18:37:58.966007 24386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.2:39483 every 8 connection(s)
I20260620 18:37:58.967178 24259 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980676253068-24012-0/minicluster-data/ts-1/data/info.pb
I20260620 18:37:58.969622 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 24259
I20260620 18:37:58.969818 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980676253068-24012-0/minicluster-data/ts-1/wal/instance
I20260620 18:37:58.970366 24012 external_mini_cluster.cc:1428] Setting key 0b4dbee347e0aac89fb58a14d791df56
Jun 20 18:37:58 dist-test-slave-d645 krb5kdc[24019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980678, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.2@KRBTEST.COM for kudu/127.23.115.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.115.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.115.3@KRBTEST.COM" created.
I20260620 18:37:58.989147 24255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:58.969836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.2:41071 (local address 127.23.115.62:32937)
0620 18:37:58.970016 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:58.970021 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:58.971420 (+ 1399us) server_negotiation.cc:407] Connection header received
0620 18:37:58.973601 (+ 2181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:58.973605 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:58.973674 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:58.973776 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:58.975966 (+ 2190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:58.976728 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:58.978374 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:58.978813 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:58.978961 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:58.982688 (+ 3727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:58.982696 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:58.982701 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:37:58.982746 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:58.985981 (+ 3235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:58.986749 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:58.986754 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:58.986756 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:58.986882 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:58.987388 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:58.987393 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:58.987395 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:58.987688 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:58.987824 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:58.988572 (+ 748us) server_negotiation.cc:299] Negotiation successful
0620 18:37:58.988801 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 18:37:58.990110 24389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:58.970366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:32937 (local address 127.23.115.2:41071)
0620 18:37:58.971249 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:58.971295 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 18:37:58.973291 (+ 1996us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:37:58.974109 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:37:58.974123 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:37:58.974874 (+ 751us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:37:58.975760 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:58.975786 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:58.976900 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:58.976907 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:58.978209 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:58.978229 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:58.979075 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:58.979083 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:58.979319 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:58.979995 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:37:58.980022 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:37:58.982393 (+ 2371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:37:58.986116 (+ 3723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:37:58.986123 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:37:58.986138 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:37:58.986617 (+ 479us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:37:58.987031 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:37:58.987035 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:37:58.987038 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:37:58.987255 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:37:58.987817 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:37:58.987825 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:37:58.988195 (+ 370us) client_negotiation.cc:769] Sending connection context
0620 18:37:58.988445 (+ 250us) client_negotiation.cc:240] Negotiation successful
0620 18:37:58.988983 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"thread_start_us":326,"threads_started":1}
I20260620 18:37:58.991701 24387 heartbeater.cc:347] Connected to a master server at 127.23.115.62:32937
I20260620 18:37:58.992038 24387 heartbeater.cc:464] Registering TS with master...
I20260620 18:37:58.992905 24387 heartbeater.cc:511] Master 127.23.115.62:32937 requested a full tablet report, sending...
I20260620 18:37:58.994439 24064 ts_manager.cc:194] Registered new tserver with Master: 4579bdedd0ae42d2b4585c694727e714 (127.23.115.2:39483)
I20260620 18:37:58.995313 24064 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.115.2@KRBTEST.COM'} at 127.23.115.2:41071
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.115.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.3.keytab.
Entry for principal kudu/127.23.115.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.3.keytab.
I20260620 18:37:59.011374 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980676253068-24012-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980676253068-24012-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980676253068-24012-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980676253068-24012-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.115.3:0
--local_ip_for_outbound_sockets=127.23.115.3
--webserver_interface=127.23.115.3
--webserver_port=0
--tserver_master_addrs=127.23.115.62:32937
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.3.keytab
--principal=kudu/127.23.115.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:37731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:37:59.328536 24392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:37:59.328817 24392 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:37:59.328914 24392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:37:59.336738 24392 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:37:59.336874 24392 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:37:59.337034 24392 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.115.3
I20260620 18:37:59.347040 24392 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:37731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980676253068-24012-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980676253068-24012-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.3.keytab
--principal=kudu/127.23.115.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.115.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980676253068-24012-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.115.3
--webserver_port=0
--tserver_master_addrs=127.23.115.62:32937
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.115.3
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980676253068-24012-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:37:59.348367 24392 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:37:59.349620 24392 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:37:59.362265 24397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:37:59.362301 24398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:37:59.362344 24400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:37:59.363304 24392 server_base.cc:1034] running on GCE node
I20260620 18:37:59.363893 24392 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:37:59.364774 24392 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:37:59.365976 24392 hybrid_clock.cc:648] HybridClock initialized: now 1781980679365933 us; error 34 us; skew 500 ppm
I20260620 18:37:59.366284 24392 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:37:59 dist-test-slave-d645 krb5kdc[24019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980679, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:37:59.372864 24392 init.cc:377] Logged in from keytab as kudu/127.23.115.3@KRBTEST.COM (short username kudu)
I20260620 18:37:59.374406 24392 webserver.cc:460] Webserver started at http://127.23.115.3:37239/ using document root <none> and password file <none>
I20260620 18:37:59.375137 24392 fs_manager.cc:359] Metadata directory not provided
I20260620 18:37:59.375244 24392 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:37:59.375500 24392 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:37:59.378262 24392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980676253068-24012-0/minicluster-data/ts-2/data/instance:
uuid: "68f82c4ccd4d4f1e8eaf2f6cea5c0194"
format_stamp: "Formatted at 2026-06-20 18:37:59 on dist-test-slave-d645"
server_key: "58289bf35e21e60e8e4bc43d079aa504"
server_key_iv: "11f4c86369686c1f4e6fe52eed87141d"
server_key_version: "encryptionkey@0"
I20260620 18:37:59.378964 24392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980676253068-24012-0/minicluster-data/ts-2/wal/instance:
uuid: "68f82c4ccd4d4f1e8eaf2f6cea5c0194"
format_stamp: "Formatted at 2026-06-20 18:37:59 on dist-test-slave-d645"
server_key: "58289bf35e21e60e8e4bc43d079aa504"
server_key_iv: "11f4c86369686c1f4e6fe52eed87141d"
server_key_version: "encryptionkey@0"
I20260620 18:37:59.383987 24392 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 18:37:59.387281 24407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:37:59.388643 24392 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 18:37:59.388788 24392 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980676253068-24012-0/minicluster-data/ts-2/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980676253068-24012-0/minicluster-data/ts-2/wal
uuid: "68f82c4ccd4d4f1e8eaf2f6cea5c0194"
format_stamp: "Formatted at 2026-06-20 18:37:59 on dist-test-slave-d645"
server_key: "58289bf35e21e60e8e4bc43d079aa504"
server_key_iv: "11f4c86369686c1f4e6fe52eed87141d"
server_key_version: "encryptionkey@0"
I20260620 18:37:59.388963 24392 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980676253068-24012-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980676253068-24012-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980676253068-24012-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:37:59.404589 24392 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:37:59.408663 24392 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:37:59.408923 24392 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:37:59.409878 24392 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:37:59.411207 24392 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:37:59.411284 24392 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:37:59.411397 24392 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:37:59.411453 24392 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:37:59.442920 24392 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.3:42831
I20260620 18:37:59.443053 24519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.3:42831 every 8 connection(s)
I20260620 18:37:59.444304 24392 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980676253068-24012-0/minicluster-data/ts-2/data/info.pb
I20260620 18:37:59.449244 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 24392
I20260620 18:37:59.449450 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980676253068-24012-0/minicluster-data/ts-2/wal/instance
I20260620 18:37:59.449971 24012 external_mini_cluster.cc:1428] Setting key 7202b1d9740bcc24a461ee172db08f2e
Jun 20 18:37:59 dist-test-slave-d645 krb5kdc[24019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980679, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.3@KRBTEST.COM for kudu/127.23.115.62@KRBTEST.COM
I20260620 18:37:59.465456 24255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:59.447032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.3:54211 (local address 127.23.115.62:32937)
0620 18:37:59.447202 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:59.447206 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:59.448902 (+ 1696us) server_negotiation.cc:407] Connection header received
0620 18:37:59.451334 (+ 2432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:59.451340 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:59.451457 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:59.451556 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:59.453394 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:59.454169 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:59.455715 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:59.456091 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:59.456203 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:59.459538 (+ 3335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:59.459543 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:59.459546 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:37:59.459580 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:59.462486 (+ 2906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:59.463234 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:59.463238 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:59.463240 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:59.463361 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:59.463761 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:59.463765 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:59.463767 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:59.464029 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:59.464162 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:59.464888 (+ 726us) server_negotiation.cc:299] Negotiation successful
0620 18:37:59.465105 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 18:37:59.466404 24522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:59.447788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:32937 (local address 127.23.115.3:54211)
0620 18:37:59.448685 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:59.448734 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 18:37:59.450990 (+ 2256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:37:59.451840 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:37:59.451852 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:37:59.452531 (+ 679us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:37:59.453212 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:59.453230 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:59.454316 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:59.454319 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:59.455578 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:59.455590 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:59.456320 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:59.456327 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:59.456554 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:59.457173 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:37:59.457197 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:37:59.459330 (+ 2133us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:37:59.462639 (+ 3309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:37:59.462646 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:37:59.462660 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:37:59.463107 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:37:59.463463 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:37:59.463466 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:37:59.463469 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:37:59.463650 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:37:59.464166 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:37:59.464175 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:37:59.464515 (+ 340us) client_negotiation.cc:769] Sending connection context
0620 18:37:59.464752 (+ 237us) client_negotiation.cc:240] Negotiation successful
0620 18:37:59.465270 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"thread_start_us":416,"threads_started":1}
I20260620 18:37:59.467677 24520 heartbeater.cc:347] Connected to a master server at 127.23.115.62:32937
I20260620 18:37:59.467948 24520 heartbeater.cc:464] Registering TS with master...
I20260620 18:37:59.468641 24520 heartbeater.cc:511] Master 127.23.115.62:32937 requested a full tablet report, sending...
I20260620 18:37:59.470058 24064 ts_manager.cc:194] Registered new tserver with Master: 68f82c4ccd4d4f1e8eaf2f6cea5c0194 (127.23.115.3:42831)
I20260620 18:37:59.470822 24064 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.115.3@KRBTEST.COM'} at 127.23.115.3:54211
I20260620 18:37:59.478437 24012 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 18:37:59 dist-test-slave-d645 krb5kdc[24019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980679, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 18:37:59 dist-test-slave-d645 krb5kdc[24019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980679, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.115.1@KRBTEST.COM
I20260620 18:37:59.521736 24527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:59.506008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34326 (local address 127.23.115.1:36651)
0620 18:37:59.506586 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:59.506594 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:59.506623 (+ 29us) server_negotiation.cc:407] Connection header received
0620 18:37:59.506779 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:59.506786 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:59.507018 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:59.507159 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:59.508573 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:59.509567 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:59.510885 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:59.511510 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:59.511624 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:59.515572 (+ 3948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:59.515585 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:59.515599 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:37:59.515638 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:59.518427 (+ 2789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:59.519179 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:59.519187 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:59.519189 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:59.519327 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:59.519870 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:59.519877 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:59.519879 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:59.520553 (+ 674us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:59.520847 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:59.521109 (+ 262us) server_negotiation.cc:299] Negotiation successful
0620 18:37:59.521321 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":301,"threads_started":1}
W20260620 18:37:59.522956 24168 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34326
I20260620 18:37:59.527169 24253 heartbeater.cc:503] Master 127.23.115.62:32937 was elected leader, sending a full tablet report...
Jun 20 18:37:59 dist-test-slave-d645 krb5kdc[24019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 18:37:59 dist-test-slave-d645 krb5kdc[24019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.115.1@KRBTEST.COM
I20260620 18:37:59.562857 24527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:59.549802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34328 (local address 127.23.115.1:36651)
0620 18:37:59.550001 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:59.550009 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:59.550298 (+ 289us) server_negotiation.cc:407] Connection header received
0620 18:37:59.550472 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:59.550476 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:59.550570 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:59.550656 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:59.551958 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:59.552703 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:59.553871 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:59.554294 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:59.554400 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:59.558022 (+ 3622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:59.558026 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:59.558029 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:37:59.558062 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:59.560509 (+ 2447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:59.561103 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:59.561108 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:59.561111 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:59.561232 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:59.561570 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:59.561575 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:59.561579 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:59.561825 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:59.562039 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:59.562340 (+ 301us) server_negotiation.cc:299] Negotiation successful
0620 18:37:59.562546 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 18:37:59.564280 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 24125
I20260620 18:37:59.575990 24527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:59.565175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34342 (local address 127.23.115.1:36651)
0620 18:37:59.565476 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:59.565481 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:59.565695 (+ 214us) server_negotiation.cc:407] Connection header received
0620 18:37:59.565772 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:59.565775 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:59.565833 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:59.565925 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:59.567153 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:59.567846 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:59.569085 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:59.569518 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:59.569610 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:59.571668 (+ 2058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:59.571673 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:59.571676 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:37:59.571703 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:59.573929 (+ 2226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:59.574525 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:59.574529 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:59.574531 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:59.574648 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:59.574923 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:59.574926 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:59.574973 (+ 47us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:59.575209 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:59.575380 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:59.575583 (+ 203us) server_negotiation.cc:299] Negotiation successful
0620 18:37:59.575725 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260620 18:37:59.576633 24248 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:37:59.658042 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 24125
I20260620 18:37:59.670712 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 24259
Jun 20 18:37:59 dist-test-slave-d645 krb5kdc[24019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.115.2@KRBTEST.COM
I20260620 18:37:59.686049 24534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:59.671582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48382 (local address 127.23.115.2:39483)
0620 18:37:59.672125 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:59.672132 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:59.672164 (+ 32us) server_negotiation.cc:407] Connection header received
0620 18:37:59.672221 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:59.672227 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:59.672440 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:59.672577 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:59.673734 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:59.674710 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:59.676064 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:59.676605 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:59.676728 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:59.680644 (+ 3916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:59.680657 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:59.680672 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:37:59.680707 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:59.683413 (+ 2706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:59.684042 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:59.684049 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:59.684051 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:59.684167 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:59.684557 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:59.684561 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:59.684563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:59.684888 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:59.685192 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:59.685454 (+ 262us) server_negotiation.cc:299] Negotiation successful
0620 18:37:59.685710 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":235,"threads_started":1}
I20260620 18:37:59.687364 24382 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:37:59.769418 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 24259
I20260620 18:37:59.781637 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 24392
Jun 20 18:37:59 dist-test-slave-d645 krb5kdc[24019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.115.3@KRBTEST.COM
I20260620 18:37:59.798058 24536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:59.782530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42796 (local address 127.23.115.3:42831)
0620 18:37:59.783116 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:59.783125 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:59.783155 (+ 30us) server_negotiation.cc:407] Connection header received
0620 18:37:59.783215 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:59.783221 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:59.783428 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:59.783562 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:59.785012 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:59.786063 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:59.787599 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:59.788190 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:59.788303 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:59.792394 (+ 4091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:59.792406 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:59.792421 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:37:59.792456 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:59.795339 (+ 2883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:59.796038 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:59.796044 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:59.796047 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:59.796167 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:59.796513 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:59.796517 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:59.796519 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:59.796870 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:59.797163 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:59.797474 (+ 311us) server_negotiation.cc:299] Negotiation successful
0620 18:37:59.797738 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":273,"threads_started":1}
I20260620 18:37:59.799316 24515 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:37:59.880913 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 24392
I20260620 18:37:59.893231 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 24032
I20260620 18:37:59.894399 24094 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:37:59.994831 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 24032
2026-06-20T18:38:00Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3708 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980676253068-24012-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:38:00 dist-test-slave-d645 krb5kdc[24541](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:38:00 dist-test-slave-d645 krb5kdc[24541](info): set up 2 sockets
Jun 20 18:38:00 dist-test-slave-d645 krb5kdc[24541](info): commencing operation
krb5kdc: starting...
W20260620 18:38:01.007413 24012 mini_kdc.cc:121] Time spent starting KDC: real 0.972s user 0.002s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980676253068-24012-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980676253068-24012-0/krb5kdc/test-user.keytab.
Jun 20 18:38:01 dist-test-slave-d645 krb5kdc[24541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980681, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:38:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:38:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.115.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.115.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.115.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab.
Entry for principal kudu/127.23.115.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab.
I20260620 18:38:01.179450 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980676253068-24012-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980676253068-24012-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980676253068-24012-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.115.62:46161
--webserver_interface=127.23.115.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab
--principal=kudu/127.23.115.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:46605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.115.62:46161
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:38:01.491111 24554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:01.491392 24554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:01.491479 24554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:01.499012 24554 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:38:01.499127 24554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:01.499182 24554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:01.499248 24554 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:38:01.499289 24554 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:38:01.509910 24554 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:46605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980676253068-24012-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980676253068-24012-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.115.62:46161
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab
--principal=kudu/127.23.115.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.115.62:46161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.115.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980676253068-24012-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:38:01.511196 24554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:01.512434 24554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:01.524498 24559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:01.524454 24562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:01.524529 24560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:01.525605 24554 server_base.cc:1034] running on GCE node
I20260620 18:38:01.526365 24554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:01.527823 24554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:01.529039 24554 hybrid_clock.cc:648] HybridClock initialized: now 1781980681528983 us; error 46 us; skew 500 ppm
I20260620 18:38:01.529397 24554 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:38:01 dist-test-slave-d645 krb5kdc[24541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980681, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:01.532409 24554 init.cc:377] Logged in from keytab as kudu/127.23.115.62@KRBTEST.COM (short username kudu)
I20260620 18:38:01.533914 24554 webserver.cc:460] Webserver started at http://127.23.115.62:36547/ using document root <none> and password file <none>
I20260620 18:38:01.534578 24554 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:01.534789 24554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:01.535203 24554 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:38:01.538165 24554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980676253068-24012-0/minicluster-data/master-0/data/instance:
uuid: "96a391c5643e47918c781b0fef8cae2a"
format_stamp: "Formatted at 2026-06-20 18:38:01 on dist-test-slave-d645"
server_key: "745d9af69ac78640c49d00d656ffe122"
server_key_iv: "d6f0d0809f109aaa3ce811cb6ac865bc"
server_key_version: "encryptionkey@0"
I20260620 18:38:01.539237 24554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980676253068-24012-0/minicluster-data/master-0/wal/instance:
uuid: "96a391c5643e47918c781b0fef8cae2a"
format_stamp: "Formatted at 2026-06-20 18:38:01 on dist-test-slave-d645"
server_key: "745d9af69ac78640c49d00d656ffe122"
server_key_iv: "d6f0d0809f109aaa3ce811cb6ac865bc"
server_key_version: "encryptionkey@0"
I20260620 18:38:01.544893 24554 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:38:01.548583 24569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:01.550079 24554 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:38:01.550257 24554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980676253068-24012-0/minicluster-data/master-0/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980676253068-24012-0/minicluster-data/master-0/wal
uuid: "96a391c5643e47918c781b0fef8cae2a"
format_stamp: "Formatted at 2026-06-20 18:38:01 on dist-test-slave-d645"
server_key: "745d9af69ac78640c49d00d656ffe122"
server_key_iv: "d6f0d0809f109aaa3ce811cb6ac865bc"
server_key_version: "encryptionkey@0"
I20260620 18:38:01.550447 24554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980676253068-24012-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980676253068-24012-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980676253068-24012-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:01.579741 24554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:01.585224 24554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:01.585517 24554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:01.604143 24554 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.62:46161
I20260620 18:38:01.604146 24620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.62:46161 every 8 connection(s)
I20260620 18:38:01.605628 24554 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
I20260620 18:38:01.609664 24621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:01.615778 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 24554
I20260620 18:38:01.616099 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980676253068-24012-0/minicluster-data/master-0/wal/instance
I20260620 18:38:01.616837 24012 external_mini_cluster.cc:1428] Setting key 5e77b0dcb0edac6aeeb72afc7cd5cb08
I20260620 18:38:01.620633 24621 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96a391c5643e47918c781b0fef8cae2a: Bootstrap starting.
I20260620 18:38:01.624542 24621 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 96a391c5643e47918c781b0fef8cae2a: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:01.625777 24621 log.cc:824] T 00000000000000000000000000000000 P 96a391c5643e47918c781b0fef8cae2a: Log is configured to *not* fsync() on all Append() calls
I20260620 18:38:01.628682 24621 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96a391c5643e47918c781b0fef8cae2a: No bootstrap required, opened a new log
Jun 20 18:38:01 dist-test-slave-d645 krb5kdc[24541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980681, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.115.62@KRBTEST.COM
I20260620 18:38:01.638569 24621 raft_consensus.cc:348] T 00000000000000000000000000000000 P 96a391c5643e47918c781b0fef8cae2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96a391c5643e47918c781b0fef8cae2a" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 46161 } }
I20260620 18:38:01.639068 24621 raft_consensus.cc:374] T 00000000000000000000000000000000 P 96a391c5643e47918c781b0fef8cae2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:01.639221 24621 raft_consensus.cc:729] T 00000000000000000000000000000000 P 96a391c5643e47918c781b0fef8cae2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96a391c5643e47918c781b0fef8cae2a, State: Initialized, Role: FOLLOWER
I20260620 18:38:01.639921 24621 consensus_queue.cc:260] T 00000000000000000000000000000000 P 96a391c5643e47918c781b0fef8cae2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96a391c5643e47918c781b0fef8cae2a" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 46161 } }
I20260620 18:38:01.640230 24621 raft_consensus.cc:388] T 00000000000000000000000000000000 P 96a391c5643e47918c781b0fef8cae2a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:38:01.640364 24621 raft_consensus.cc:482] T 00000000000000000000000000000000 P 96a391c5643e47918c781b0fef8cae2a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:38:01.640537 24621 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 96a391c5643e47918c781b0fef8cae2a [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:01.642294 24624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:01.619204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51850 (local address 127.23.115.62:46161)
0620 18:38:01.620033 (+ 829us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:01.620050 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:01.620101 (+ 51us) server_negotiation.cc:407] Connection header received
0620 18:38:01.622518 (+ 2417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:01.622547 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:01.623113 (+ 566us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:01.623548 (+ 435us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:01.625010 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:01.626802 (+ 1792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:01.628078 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:01.628744 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:01.628949 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:01.633620 (+ 4671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:01.633631 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:01.633665 (+ 34us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:01.633717 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:01.637759 (+ 4042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:01.638496 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:01.638504 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:01.638513 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:01.638755 (+ 242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:01.639185 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:01.639190 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:01.639193 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:01.639926 (+ 733us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:01.640230 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:01.640794 (+ 564us) server_negotiation.cc:299] Negotiation successful
0620 18:38:01.641242 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":339,"threads_started":1}
I20260620 18:38:01.643028 24621 raft_consensus.cc:504] T 00000000000000000000000000000000 P 96a391c5643e47918c781b0fef8cae2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96a391c5643e47918c781b0fef8cae2a" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 46161 } }
I20260620 18:38:01.643507 24621 leader_election.cc:302] T 00000000000000000000000000000000 P 96a391c5643e47918c781b0fef8cae2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 96a391c5643e47918c781b0fef8cae2a; no voters:
I20260620 18:38:01.644106 24621 leader_election.cc:288] T 00000000000000000000000000000000 P 96a391c5643e47918c781b0fef8cae2a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:38:01.644213 24626 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 96a391c5643e47918c781b0fef8cae2a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:38:01.645569 24626 raft_consensus.cc:686] T 00000000000000000000000000000000 P 96a391c5643e47918c781b0fef8cae2a [term 1 LEADER]: Becoming Leader. State: Replica: 96a391c5643e47918c781b0fef8cae2a, State: Running, Role: LEADER
I20260620 18:38:01.646104 24621 sys_catalog.cc:564] T 00000000000000000000000000000000 P 96a391c5643e47918c781b0fef8cae2a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:38:01.646055 24626 consensus_queue.cc:237] T 00000000000000000000000000000000 P 96a391c5643e47918c781b0fef8cae2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96a391c5643e47918c781b0fef8cae2a" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 46161 } }
I20260620 18:38:01.648729 24628 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96a391c5643e47918c781b0fef8cae2a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 96a391c5643e47918c781b0fef8cae2a. Latest consensus state: current_term: 1 leader_uuid: "96a391c5643e47918c781b0fef8cae2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96a391c5643e47918c781b0fef8cae2a" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 46161 } } }
I20260620 18:38:01.649219 24628 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96a391c5643e47918c781b0fef8cae2a [sys.catalog]: This master's current role is: LEADER
I20260620 18:38:01.649641 24627 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96a391c5643e47918c781b0fef8cae2a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "96a391c5643e47918c781b0fef8cae2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96a391c5643e47918c781b0fef8cae2a" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 46161 } } }
I20260620 18:38:01.649892 24627 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96a391c5643e47918c781b0fef8cae2a [sys.catalog]: This master's current role is: LEADER
I20260620 18:38:01.650774 24635 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:38:01.656771 24635 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:38:01.664901 24635 catalog_manager.cc:1347] Generated new cluster ID: 3d50ce1707164a689b6c47cec72a6ac1
I20260620 18:38:01.665020 24635 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:38:01.679580 24635 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:38:01.681481 24635 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:38:01.695955 24635 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 96a391c5643e47918c781b0fef8cae2a: Generated new TSK 0
I20260620 18:38:01.697101 24635 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.115.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.115.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.115.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.1.keytab.
Entry for principal kudu/127.23.115.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.1.keytab.
I20260620 18:38:01.744444 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980676253068-24012-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980676253068-24012-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980676253068-24012-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980676253068-24012-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.115.1:0
--local_ip_for_outbound_sockets=127.23.115.1
--webserver_interface=127.23.115.1
--webserver_port=0
--tserver_master_addrs=127.23.115.62:46161
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.1.keytab
--principal=kudu/127.23.115.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:46605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:38:02.057591 24647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:02.057857 24647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:02.057952 24647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:02.065088 24647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:02.065191 24647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:02.065434 24647 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.115.1
I20260620 18:38:02.075152 24647 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:46605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980676253068-24012-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980676253068-24012-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.1.keytab
--principal=kudu/127.23.115.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.115.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980676253068-24012-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.115.1
--webserver_port=0
--tserver_master_addrs=127.23.115.62:46161
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.115.1
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980676253068-24012-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:38:02.076368 24647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:02.077610 24647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:02.090651 24652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:02.090739 24653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:02.090705 24655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:02.091295 24647 server_base.cc:1034] running on GCE node
I20260620 18:38:02.091909 24647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:02.092792 24647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:02.094010 24647 hybrid_clock.cc:648] HybridClock initialized: now 1781980682093959 us; error 35 us; skew 500 ppm
I20260620 18:38:02.094321 24647 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:38:02 dist-test-slave-d645 krb5kdc[24541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980682, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:02.097368 24647 init.cc:377] Logged in from keytab as kudu/127.23.115.1@KRBTEST.COM (short username kudu)
I20260620 18:38:02.098850 24647 webserver.cc:460] Webserver started at http://127.23.115.1:42583/ using document root <none> and password file <none>
I20260620 18:38:02.099536 24647 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:02.099654 24647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:02.099927 24647 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:38:02.102705 24647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980676253068-24012-0/minicluster-data/ts-0/data/instance:
uuid: "bb794b381e7242cb9eda42990aa23a55"
format_stamp: "Formatted at 2026-06-20 18:38:02 on dist-test-slave-d645"
server_key: "11b1bdbbe3c7f80d76d91a6e20c3fa29"
server_key_iv: "b6a8e20473854b608cf5a3f014282146"
server_key_version: "encryptionkey@0"
I20260620 18:38:02.103672 24647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980676253068-24012-0/minicluster-data/ts-0/wal/instance:
uuid: "bb794b381e7242cb9eda42990aa23a55"
format_stamp: "Formatted at 2026-06-20 18:38:02 on dist-test-slave-d645"
server_key: "11b1bdbbe3c7f80d76d91a6e20c3fa29"
server_key_iv: "b6a8e20473854b608cf5a3f014282146"
server_key_version: "encryptionkey@0"
I20260620 18:38:02.108456 24647 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 18:38:02.111821 24662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:02.113198 24647 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:38:02.113354 24647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980676253068-24012-0/minicluster-data/ts-0/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980676253068-24012-0/minicluster-data/ts-0/wal
uuid: "bb794b381e7242cb9eda42990aa23a55"
format_stamp: "Formatted at 2026-06-20 18:38:02 on dist-test-slave-d645"
server_key: "11b1bdbbe3c7f80d76d91a6e20c3fa29"
server_key_iv: "b6a8e20473854b608cf5a3f014282146"
server_key_version: "encryptionkey@0"
I20260620 18:38:02.113526 24647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980676253068-24012-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980676253068-24012-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980676253068-24012-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:02.142027 24647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:02.146130 24647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:02.146356 24647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:02.147302 24647 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:38:02.148566 24647 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:38:02.148639 24647 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:02.148716 24647 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:38:02.148758 24647 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:02.181203 24647 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.1:35773
I20260620 18:38:02.181248 24774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.1:35773 every 8 connection(s)
I20260620 18:38:02.182377 24647 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980676253068-24012-0/minicluster-data/ts-0/data/info.pb
I20260620 18:38:02.190585 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 24647
I20260620 18:38:02.190775 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980676253068-24012-0/minicluster-data/ts-0/wal/instance
I20260620 18:38:02.191411 24012 external_mini_cluster.cc:1428] Setting key 3b9b9791c9edd2275cf330440ae9d003
Jun 20 18:38:02 dist-test-slave-d645 krb5kdc[24541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980682, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.1@KRBTEST.COM for kudu/127.23.115.62@KRBTEST.COM
I20260620 18:38:02.202353 24777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:02.184754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.1:38715 (local address 127.23.115.62:46161)
0620 18:38:02.185162 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:02.185166 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:02.186064 (+ 898us) server_negotiation.cc:407] Connection header received
0620 18:38:02.187846 (+ 1782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:02.187851 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:02.187923 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:02.188019 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:02.189424 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:02.190207 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:02.191846 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:02.192306 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:02.192432 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:02.196240 (+ 3808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:02.196246 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:02.196251 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:02.196289 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:02.198991 (+ 2702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:02.200107 (+ 1116us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:02.200111 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:02.200113 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:02.200251 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:02.200683 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:02.200689 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:02.200691 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:02.200961 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:02.201086 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:02.201881 (+ 795us) server_negotiation.cc:299] Negotiation successful
0620 18:38:02.202082 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":197,"threads_started":1}
I20260620 18:38:02.203647 24778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:02.185224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:46161 (local address 127.23.115.1:38715)
0620 18:38:02.185887 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:02.185929 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:02.187598 (+ 1669us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:02.188203 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:02.188210 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:02.188677 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:02.189254 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:02.189271 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:02.190345 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:02.190349 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:02.191709 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:02.191722 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:02.192585 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:02.192594 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:02.192889 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:02.193609 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:02.193633 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:02.195981 (+ 2348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:02.199361 (+ 3380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:02.199370 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:02.199386 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:02.199957 (+ 571us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:02.200348 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:02.200353 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:02.200356 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:02.200576 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:02.201071 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:02.201079 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:02.201486 (+ 407us) client_negotiation.cc:769] Sending connection context
0620 18:38:02.201745 (+ 259us) client_negotiation.cc:240] Negotiation successful
0620 18:38:02.202285 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":263,"threads_started":1}
I20260620 18:38:02.205291 24775 heartbeater.cc:347] Connected to a master server at 127.23.115.62:46161
I20260620 18:38:02.205586 24775 heartbeater.cc:464] Registering TS with master...
I20260620 18:38:02.206283 24775 heartbeater.cc:511] Master 127.23.115.62:46161 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.23.115.2@KRBTEST.COM; defaulting to no policy
I20260620 18:38:02.208333 24586 ts_manager.cc:194] Registered new tserver with Master: bb794b381e7242cb9eda42990aa23a55 (127.23.115.1:35773)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.115.2@KRBTEST.COM" created.
I20260620 18:38:02.209918 24586 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.115.1@KRBTEST.COM'} at 127.23.115.1:38715
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.115.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.2.keytab.
Entry for principal kudu/127.23.115.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.2.keytab.
I20260620 18:38:02.230259 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980676253068-24012-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980676253068-24012-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980676253068-24012-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980676253068-24012-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.115.2:0
--local_ip_for_outbound_sockets=127.23.115.2
--webserver_interface=127.23.115.2
--webserver_port=0
--tserver_master_addrs=127.23.115.62:46161
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.2.keytab
--principal=kudu/127.23.115.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:46605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:38:02.533206 24781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:02.533493 24781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:02.533630 24781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:02.540723 24781 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:02.540849 24781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:02.541008 24781 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.115.2
I20260620 18:38:02.551429 24781 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:46605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980676253068-24012-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980676253068-24012-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.2.keytab
--principal=kudu/127.23.115.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.115.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980676253068-24012-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.115.2
--webserver_port=0
--tserver_master_addrs=127.23.115.62:46161
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.115.2
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980676253068-24012-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:38:02.552722 24781 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:02.554051 24781 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:02.566641 24787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:02.567014 24789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:02.567107 24786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:02.568504 24781 server_base.cc:1034] running on GCE node
I20260620 18:38:02.569114 24781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:02.570001 24781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:02.571200 24781 hybrid_clock.cc:648] HybridClock initialized: now 1781980682571142 us; error 48 us; skew 500 ppm
I20260620 18:38:02.571502 24781 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:38:02 dist-test-slave-d645 krb5kdc[24541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980682, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:02.574508 24781 init.cc:377] Logged in from keytab as kudu/127.23.115.2@KRBTEST.COM (short username kudu)
I20260620 18:38:02.575963 24781 webserver.cc:460] Webserver started at http://127.23.115.2:33785/ using document root <none> and password file <none>
I20260620 18:38:02.576623 24781 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:02.576718 24781 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:02.576968 24781 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:38:02.579840 24781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980676253068-24012-0/minicluster-data/ts-1/data/instance:
uuid: "5420bd0ecc8643f1b62c4d0aa357ab15"
format_stamp: "Formatted at 2026-06-20 18:38:02 on dist-test-slave-d645"
server_key: "3297af1ba0477673a5288afe56c261ae"
server_key_iv: "aee7337b6e25319ca5d2a1749934f3b7"
server_key_version: "encryptionkey@0"
I20260620 18:38:02.580540 24781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980676253068-24012-0/minicluster-data/ts-1/wal/instance:
uuid: "5420bd0ecc8643f1b62c4d0aa357ab15"
format_stamp: "Formatted at 2026-06-20 18:38:02 on dist-test-slave-d645"
server_key: "3297af1ba0477673a5288afe56c261ae"
server_key_iv: "aee7337b6e25319ca5d2a1749934f3b7"
server_key_version: "encryptionkey@0"
I20260620 18:38:02.585551 24781 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:38:02.589035 24796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:02.590350 24781 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:38:02.590502 24781 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980676253068-24012-0/minicluster-data/ts-1/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980676253068-24012-0/minicluster-data/ts-1/wal
uuid: "5420bd0ecc8643f1b62c4d0aa357ab15"
format_stamp: "Formatted at 2026-06-20 18:38:02 on dist-test-slave-d645"
server_key: "3297af1ba0477673a5288afe56c261ae"
server_key_iv: "aee7337b6e25319ca5d2a1749934f3b7"
server_key_version: "encryptionkey@0"
I20260620 18:38:02.590713 24781 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980676253068-24012-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980676253068-24012-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980676253068-24012-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:02.606775 24781 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:02.611066 24781 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:02.611328 24781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:02.612254 24781 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:38:02.613547 24781 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:38:02.613620 24781 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:02.613740 24781 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:38:02.613790 24781 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:02.643724 24781 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.2:37059
I20260620 18:38:02.643738 24908 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.2:37059 every 8 connection(s)
I20260620 18:38:02.644901 24781 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980676253068-24012-0/minicluster-data/ts-1/data/info.pb
I20260620 18:38:02.645294 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 24781
I20260620 18:38:02.645466 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980676253068-24012-0/minicluster-data/ts-1/wal/instance
I20260620 18:38:02.646054 24012 external_mini_cluster.cc:1428] Setting key 18bd85318a6d5c598f02a0d47ce84b84
Jun 20 18:38:02 dist-test-slave-d645 krb5kdc[24541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980682, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.2@KRBTEST.COM for kudu/127.23.115.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.115.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.115.3@KRBTEST.COM" created.
I20260620 18:38:02.666162 24777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:02.648008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.2:44331 (local address 127.23.115.62:46161)
0620 18:38:02.648185 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:02.648190 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:02.648997 (+ 807us) server_negotiation.cc:407] Connection header received
0620 18:38:02.650779 (+ 1782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:02.650785 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:02.650873 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:02.651012 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:02.652504 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:02.653244 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:02.655447 (+ 2203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:02.655853 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:02.655947 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:02.660492 (+ 4545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:02.660498 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:02.660501 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:02.660537 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:02.663221 (+ 2684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:02.663912 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:02.663917 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:02.663919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:02.664043 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:02.664406 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:02.664410 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:02.664414 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:02.664778 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:02.664919 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:02.665595 (+ 676us) server_negotiation.cc:299] Negotiation successful
0620 18:38:02.665821 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 18:38:02.666972 24911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:02.648028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:46161 (local address 127.23.115.2:44331)
0620 18:38:02.648844 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:02.648894 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:02.650517 (+ 1623us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:02.651183 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:02.651191 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:02.651715 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:02.652315 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:02.652339 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:02.653390 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:02.653395 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:02.655315 (+ 1920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:02.655330 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:02.656066 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:02.656072 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:02.656334 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:02.656988 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:02.657024 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:02.659760 (+ 2736us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:02.663354 (+ 3594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:02.663361 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:02.663375 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:02.663803 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:02.664127 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:02.664130 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:02.664132 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:02.664317 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:02.664898 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:02.664904 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:02.665238 (+ 334us) client_negotiation.cc:769] Sending connection context
0620 18:38:02.665466 (+ 228us) client_negotiation.cc:240] Negotiation successful
0620 18:38:02.665858 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":336,"threads_started":1}
I20260620 18:38:02.668399 24909 heartbeater.cc:347] Connected to a master server at 127.23.115.62:46161
I20260620 18:38:02.668694 24909 heartbeater.cc:464] Registering TS with master...
I20260620 18:38:02.669456 24909 heartbeater.cc:511] Master 127.23.115.62:46161 requested a full tablet report, sending...
I20260620 18:38:02.671034 24586 ts_manager.cc:194] Registered new tserver with Master: 5420bd0ecc8643f1b62c4d0aa357ab15 (127.23.115.2:37059)
I20260620 18:38:02.672056 24586 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.115.2@KRBTEST.COM'} at 127.23.115.2:44331
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.115.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.3.keytab.
Entry for principal kudu/127.23.115.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.3.keytab.
I20260620 18:38:02.686456 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980676253068-24012-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980676253068-24012-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980676253068-24012-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980676253068-24012-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.115.3:0
--local_ip_for_outbound_sockets=127.23.115.3
--webserver_interface=127.23.115.3
--webserver_port=0
--tserver_master_addrs=127.23.115.62:46161
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.3.keytab
--principal=kudu/127.23.115.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:46605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:38:02.991025 24914 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:02.991284 24914 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:02.991380 24914 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:02.998644 24914 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:02.998764 24914 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:02.998910 24914 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.115.3
I20260620 18:38:03.010324 24914 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:46605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980676253068-24012-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980676253068-24012-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.3.keytab
--principal=kudu/127.23.115.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.115.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980676253068-24012-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.115.3
--webserver_port=0
--tserver_master_addrs=127.23.115.62:46161
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.115.3
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980676253068-24012-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:38:03.011677 24914 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:03.013239 24914 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:03.027410 24920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:03.027354 24922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:03.027424 24919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:03.028563 24914 server_base.cc:1034] running on GCE node
I20260620 18:38:03.029192 24914 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:03.030145 24914 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:03.031378 24914 hybrid_clock.cc:648] HybridClock initialized: now 1781980683031326 us; error 41 us; skew 500 ppm
I20260620 18:38:03.031724 24914 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:38:03 dist-test-slave-d645 krb5kdc[24541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980683, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:03.035187 24914 init.cc:377] Logged in from keytab as kudu/127.23.115.3@KRBTEST.COM (short username kudu)
I20260620 18:38:03.036829 24914 webserver.cc:460] Webserver started at http://127.23.115.3:33183/ using document root <none> and password file <none>
I20260620 18:38:03.037492 24914 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:03.037606 24914 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:03.037904 24914 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:38:03.040777 24914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980676253068-24012-0/minicluster-data/ts-2/data/instance:
uuid: "4dbd4eade5084e3cb512f2e9b2b9c87c"
format_stamp: "Formatted at 2026-06-20 18:38:03 on dist-test-slave-d645"
server_key: "d7cc598fa867a06d7b0ee3b53e45c53b"
server_key_iv: "d10267bafa3e2c8dad32cf3af0789925"
server_key_version: "encryptionkey@0"
I20260620 18:38:03.041514 24914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980676253068-24012-0/minicluster-data/ts-2/wal/instance:
uuid: "4dbd4eade5084e3cb512f2e9b2b9c87c"
format_stamp: "Formatted at 2026-06-20 18:38:03 on dist-test-slave-d645"
server_key: "d7cc598fa867a06d7b0ee3b53e45c53b"
server_key_iv: "d10267bafa3e2c8dad32cf3af0789925"
server_key_version: "encryptionkey@0"
I20260620 18:38:03.046531 24914 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 18:38:03.050031 24929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:03.051544 24914 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:38:03.051684 24914 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980676253068-24012-0/minicluster-data/ts-2/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980676253068-24012-0/minicluster-data/ts-2/wal
uuid: "4dbd4eade5084e3cb512f2e9b2b9c87c"
format_stamp: "Formatted at 2026-06-20 18:38:03 on dist-test-slave-d645"
server_key: "d7cc598fa867a06d7b0ee3b53e45c53b"
server_key_iv: "d10267bafa3e2c8dad32cf3af0789925"
server_key_version: "encryptionkey@0"
I20260620 18:38:03.051868 24914 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980676253068-24012-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980676253068-24012-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980676253068-24012-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:03.087702 24914 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:03.091792 24914 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:03.092039 24914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:03.092955 24914 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:38:03.094295 24914 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:38:03.094370 24914 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:03.094481 24914 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:38:03.094529 24914 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:03.126199 24914 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.3:37675
I20260620 18:38:03.126256 25041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.3:37675 every 8 connection(s)
I20260620 18:38:03.127708 24914 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980676253068-24012-0/minicluster-data/ts-2/data/info.pb
I20260620 18:38:03.132529 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 24914
I20260620 18:38:03.132721 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980676253068-24012-0/minicluster-data/ts-2/wal/instance
I20260620 18:38:03.133239 24012 external_mini_cluster.cc:1428] Setting key fde673a5824d8a475124c99f146fef11
Jun 20 18:38:03 dist-test-slave-d645 krb5kdc[24541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980683, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.3@KRBTEST.COM for kudu/127.23.115.62@KRBTEST.COM
I20260620 18:38:03.147290 24777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:03.130111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.3:39817 (local address 127.23.115.62:46161)
0620 18:38:03.130308 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:03.130314 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:03.131398 (+ 1084us) server_negotiation.cc:407] Connection header received
0620 18:38:03.133534 (+ 2136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:03.133539 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:03.133623 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:03.133733 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:03.135348 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:03.136193 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:03.138210 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:03.138600 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:03.138702 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:03.141892 (+ 3190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:03.141897 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:03.141901 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:03.141932 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:03.144420 (+ 2488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:03.145058 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:03.145061 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:03.145063 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:03.145187 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:03.145600 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:03.145603 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:03.145605 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:03.145847 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:03.145948 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:03.146720 (+ 772us) server_negotiation.cc:299] Negotiation successful
0620 18:38:03.146903 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 18:38:03.148375 25044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:03.130580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:46161 (local address 127.23.115.3:39817)
0620 18:38:03.131248 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:03.131292 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:03.133247 (+ 1955us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:03.133927 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:03.133935 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:03.134487 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:03.135198 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:03.135223 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:03.136450 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:03.136456 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:03.138066 (+ 1610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:03.138082 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:03.138816 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:03.138822 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:03.139074 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:03.139651 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:03.139686 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:03.141704 (+ 2018us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:03.144544 (+ 2840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:03.144550 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:03.144560 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:03.144968 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:03.145329 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:03.145333 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:03.145336 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:03.145513 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:03.145935 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:03.145940 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:03.146306 (+ 366us) client_negotiation.cc:769] Sending connection context
0620 18:38:03.146613 (+ 307us) client_negotiation.cc:240] Negotiation successful
0620 18:38:03.147133 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":283,"threads_started":1}
I20260620 18:38:03.149523 25042 heartbeater.cc:347] Connected to a master server at 127.23.115.62:46161
I20260620 18:38:03.149845 25042 heartbeater.cc:464] Registering TS with master...
I20260620 18:38:03.150512 25042 heartbeater.cc:511] Master 127.23.115.62:46161 requested a full tablet report, sending...
I20260620 18:38:03.151895 24586 ts_manager.cc:194] Registered new tserver with Master: 4dbd4eade5084e3cb512f2e9b2b9c87c (127.23.115.3:37675)
I20260620 18:38:03.152719 24586 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.115.3@KRBTEST.COM'} at 127.23.115.3:39817
I20260620 18:38:03.160775 24012 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:38:03.180550 24777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:03.167444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51860 (local address 127.23.115.62:46161)
0620 18:38:03.167701 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:03.167708 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:03.167968 (+ 260us) server_negotiation.cc:407] Connection header received
0620 18:38:03.168108 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:03.168114 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:03.168178 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:03.168261 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:03.169768 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:03.170640 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:03.172095 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:03.172659 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:03.172907 (+ 248us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:03.175151 (+ 2244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:03.175156 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:03.175160 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:03.175192 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:03.177533 (+ 2341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:03.178216 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:03.178222 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:03.178226 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:03.178393 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:03.178852 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:03.178875 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:03.178878 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:03.179248 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:03.179433 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:03.180107 (+ 674us) server_negotiation.cc:299] Negotiation successful
0620 18:38:03.180284 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 18:38:03.187788 24586 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51860:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:38:03.195132 24586 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:38:03.213382 24775 heartbeater.cc:503] Master 127.23.115.62:46161 was elected leader, sending a full tablet report...
I20260620 18:38:03.230330 25058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:03.209124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48858 (local address 127.23.115.1:35773)
0620 18:38:03.222527 (+ 13403us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:03.222549 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:03.222583 (+ 34us) server_negotiation.cc:407] Connection header received
0620 18:38:03.222686 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:03.222696 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:03.223020 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:03.223207 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:38:03.223898 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:03.225169 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:03.227113 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:03.229019 (+ 1906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:03.229172 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:03.229413 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:03.229808 (+ 395us) server_negotiation.cc:299] Negotiation successful
0620 18:38:03.230014 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13240,"thread_start_us":13100,"threads_started":1}
I20260620 18:38:03.230396 25057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:03.208430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.1:35773 (local address 127.0.0.1:48858)
0620 18:38:03.218453 (+ 10023us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:03.218476 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:03.218622 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:03.223233 (+ 4611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:03.223240 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:03.223269 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:38:03.223734 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:03.223746 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:03.225287 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:03.225292 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:03.226967 (+ 1675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:03.226980 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:03.229243 (+ 2263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:03.229250 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:03.229550 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:03.229607 (+ 57us) client_negotiation.cc:769] Sending connection context
0620 18:38:03.229749 (+ 142us) client_negotiation.cc:240] Negotiation successful
0620 18:38:03.229955 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9880,"mutex_wait_us":59,"thread_start_us":13418,"threads_started":1}
I20260620 18:38:03.231163 25056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:03.208832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.3:37675 (local address 127.0.0.1:57610)
0620 18:38:03.222156 (+ 13324us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:03.222176 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:03.222311 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:03.224129 (+ 1818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:03.224134 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:03.224159 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:38:03.224478 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:03.224489 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:03.226708 (+ 2219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:03.226713 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:03.228662 (+ 1949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:03.228674 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:03.230364 (+ 1690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:03.230372 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:03.230549 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:03.230579 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 18:38:03.230660 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 18:38:03.230782 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13173,"mutex_wait_us":48,"thread_start_us":9326,"threads_started":1}
I20260620 18:38:03.231722 25059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:03.209385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57610 (local address 127.23.115.3:37675)
0620 18:38:03.223021 (+ 13636us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:03.223028 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:03.223053 (+ 25us) server_negotiation.cc:407] Connection header received
0620 18:38:03.223142 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:03.223149 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:03.223314 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:03.224134 (+ 820us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:38:03.224705 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:03.226530 (+ 1825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:03.228815 (+ 2285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:03.230136 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:03.230907 (+ 771us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:03.231173 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:03.231243 (+ 70us) server_negotiation.cc:299] Negotiation successful
0620 18:38:03.231400 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13454,"thread_start_us":13298,"threads_started":1}
I20260620 18:38:03.232002 25054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:03.208187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.2:37059 (local address 127.0.0.1:50584)
0620 18:38:03.209062 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:03.209108 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:03.209285 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:03.223476 (+ 14191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:03.223481 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:03.223506 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:38:03.223901 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:03.223910 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:03.226542 (+ 2632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:03.226546 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:03.229497 (+ 2951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:03.229510 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:03.231273 (+ 1763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:03.231280 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:03.231430 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:03.231459 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 18:38:03.231523 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 18:38:03.231659 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"mutex_wait_us":292,"thread_start_us":446,"threads_started":1}
I20260620 18:38:03.232860 25055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:03.208325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50584 (local address 127.23.115.2:37059)
0620 18:38:03.222911 (+ 14586us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:03.222920 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:03.222969 (+ 49us) server_negotiation.cc:407] Connection header received
0620 18:38:03.223065 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:03.223072 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:03.223283 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:03.223478 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:38:03.224062 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:03.226367 (+ 2305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:03.229649 (+ 3282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:03.231048 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:03.232020 (+ 972us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:03.232197 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:03.232272 (+ 75us) server_negotiation.cc:299] Negotiation successful
0620 18:38:03.232420 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14407,"thread_start_us":15075,"threads_started":1}
I20260620 18:38:03.235004 24710 tablet_service.cc:1467] Processing CreateTablet for tablet a2b068a72a0d4b5bb40c456cee5c4a82 (DEFAULT_TABLE table=test-table [id=57819167ea4c406e81651e766b4c4042]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:38:03.235800 24977 tablet_service.cc:1467] Processing CreateTablet for tablet a2b068a72a0d4b5bb40c456cee5c4a82 (DEFAULT_TABLE table=test-table [id=57819167ea4c406e81651e766b4c4042]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:38:03.236595 24710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2b068a72a0d4b5bb40c456cee5c4a82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:03.236280 24844 tablet_service.cc:1467] Processing CreateTablet for tablet a2b068a72a0d4b5bb40c456cee5c4a82 (DEFAULT_TABLE table=test-table [id=57819167ea4c406e81651e766b4c4042]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:38:03.236855 24977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2b068a72a0d4b5bb40c456cee5c4a82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:03.237509 24844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2b068a72a0d4b5bb40c456cee5c4a82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:03.247819 25060 tablet_bootstrap.cc:492] T a2b068a72a0d4b5bb40c456cee5c4a82 P 4dbd4eade5084e3cb512f2e9b2b9c87c: Bootstrap starting.
I20260620 18:38:03.251788 25061 tablet_bootstrap.cc:492] T a2b068a72a0d4b5bb40c456cee5c4a82 P 5420bd0ecc8643f1b62c4d0aa357ab15: Bootstrap starting.
I20260620 18:38:03.252281 25060 tablet_bootstrap.cc:654] T a2b068a72a0d4b5bb40c456cee5c4a82 P 4dbd4eade5084e3cb512f2e9b2b9c87c: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:03.253316 25062 tablet_bootstrap.cc:492] T a2b068a72a0d4b5bb40c456cee5c4a82 P bb794b381e7242cb9eda42990aa23a55: Bootstrap starting.
I20260620 18:38:03.253733 25060 log.cc:824] T a2b068a72a0d4b5bb40c456cee5c4a82 P 4dbd4eade5084e3cb512f2e9b2b9c87c: Log is configured to *not* fsync() on all Append() calls
I20260620 18:38:03.255652 25061 tablet_bootstrap.cc:654] T a2b068a72a0d4b5bb40c456cee5c4a82 P 5420bd0ecc8643f1b62c4d0aa357ab15: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:03.256699 25062 tablet_bootstrap.cc:654] T a2b068a72a0d4b5bb40c456cee5c4a82 P bb794b381e7242cb9eda42990aa23a55: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:03.256912 25061 log.cc:824] T a2b068a72a0d4b5bb40c456cee5c4a82 P 5420bd0ecc8643f1b62c4d0aa357ab15: Log is configured to *not* fsync() on all Append() calls
I20260620 18:38:03.257385 25060 tablet_bootstrap.cc:492] T a2b068a72a0d4b5bb40c456cee5c4a82 P 4dbd4eade5084e3cb512f2e9b2b9c87c: No bootstrap required, opened a new log
I20260620 18:38:03.257675 25062 log.cc:824] T a2b068a72a0d4b5bb40c456cee5c4a82 P bb794b381e7242cb9eda42990aa23a55: Log is configured to *not* fsync() on all Append() calls
I20260620 18:38:03.257728 25060 ts_tablet_manager.cc:1397] T a2b068a72a0d4b5bb40c456cee5c4a82 P 4dbd4eade5084e3cb512f2e9b2b9c87c: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260620 18:38:03.260139 25061 tablet_bootstrap.cc:492] T a2b068a72a0d4b5bb40c456cee5c4a82 P 5420bd0ecc8643f1b62c4d0aa357ab15: No bootstrap required, opened a new log
I20260620 18:38:03.260403 25061 ts_tablet_manager.cc:1397] T a2b068a72a0d4b5bb40c456cee5c4a82 P 5420bd0ecc8643f1b62c4d0aa357ab15: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.002s
I20260620 18:38:03.261097 25062 tablet_bootstrap.cc:492] T a2b068a72a0d4b5bb40c456cee5c4a82 P bb794b381e7242cb9eda42990aa23a55: No bootstrap required, opened a new log
I20260620 18:38:03.261358 25062 ts_tablet_manager.cc:1397] T a2b068a72a0d4b5bb40c456cee5c4a82 P bb794b381e7242cb9eda42990aa23a55: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.007s
I20260620 18:38:03.268194 25062 raft_consensus.cc:348] T a2b068a72a0d4b5bb40c456cee5c4a82 P bb794b381e7242cb9eda42990aa23a55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5420bd0ecc8643f1b62c4d0aa357ab15" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 37059 } } peers { permanent_uuid: "bb794b381e7242cb9eda42990aa23a55" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 35773 } } peers { permanent_uuid: "4dbd4eade5084e3cb512f2e9b2b9c87c" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 37675 } }
I20260620 18:38:03.268548 25062 raft_consensus.cc:374] T a2b068a72a0d4b5bb40c456cee5c4a82 P bb794b381e7242cb9eda42990aa23a55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:03.268656 25062 raft_consensus.cc:729] T a2b068a72a0d4b5bb40c456cee5c4a82 P bb794b381e7242cb9eda42990aa23a55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb794b381e7242cb9eda42990aa23a55, State: Initialized, Role: FOLLOWER
I20260620 18:38:03.268800 25060 raft_consensus.cc:348] T a2b068a72a0d4b5bb40c456cee5c4a82 P 4dbd4eade5084e3cb512f2e9b2b9c87c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5420bd0ecc8643f1b62c4d0aa357ab15" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 37059 } } peers { permanent_uuid: "bb794b381e7242cb9eda42990aa23a55" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 35773 } } peers { permanent_uuid: "4dbd4eade5084e3cb512f2e9b2b9c87c" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 37675 } }
I20260620 18:38:03.269215 25060 raft_consensus.cc:374] T a2b068a72a0d4b5bb40c456cee5c4a82 P 4dbd4eade5084e3cb512f2e9b2b9c87c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:03.269311 25060 raft_consensus.cc:729] T a2b068a72a0d4b5bb40c456cee5c4a82 P 4dbd4eade5084e3cb512f2e9b2b9c87c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dbd4eade5084e3cb512f2e9b2b9c87c, State: Initialized, Role: FOLLOWER
I20260620 18:38:03.269378 25062 consensus_queue.cc:260] T a2b068a72a0d4b5bb40c456cee5c4a82 P bb794b381e7242cb9eda42990aa23a55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5420bd0ecc8643f1b62c4d0aa357ab15" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 37059 } } peers { permanent_uuid: "bb794b381e7242cb9eda42990aa23a55" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 35773 } } peers { permanent_uuid: "4dbd4eade5084e3cb512f2e9b2b9c87c" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 37675 } }
I20260620 18:38:03.269856 25060 consensus_queue.cc:260] T a2b068a72a0d4b5bb40c456cee5c4a82 P 4dbd4eade5084e3cb512f2e9b2b9c87c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5420bd0ecc8643f1b62c4d0aa357ab15" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 37059 } } peers { permanent_uuid: "bb794b381e7242cb9eda42990aa23a55" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 35773 } } peers { permanent_uuid: "4dbd4eade5084e3cb512f2e9b2b9c87c" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 37675 } }
I20260620 18:38:03.271296 25062 ts_tablet_manager.cc:1428] T a2b068a72a0d4b5bb40c456cee5c4a82 P bb794b381e7242cb9eda42990aa23a55: Time spent starting tablet: real 0.010s user 0.005s sys 0.005s
I20260620 18:38:03.272374 25042 heartbeater.cc:503] Master 127.23.115.62:46161 was elected leader, sending a full tablet report...
I20260620 18:38:03.272998 25060 ts_tablet_manager.cc:1428] T a2b068a72a0d4b5bb40c456cee5c4a82 P 4dbd4eade5084e3cb512f2e9b2b9c87c: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260620 18:38:03.273921 25061 raft_consensus.cc:348] T a2b068a72a0d4b5bb40c456cee5c4a82 P 5420bd0ecc8643f1b62c4d0aa357ab15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5420bd0ecc8643f1b62c4d0aa357ab15" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 37059 } } peers { permanent_uuid: "bb794b381e7242cb9eda42990aa23a55" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 35773 } } peers { permanent_uuid: "4dbd4eade5084e3cb512f2e9b2b9c87c" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 37675 } }
I20260620 18:38:03.274396 25061 raft_consensus.cc:374] T a2b068a72a0d4b5bb40c456cee5c4a82 P 5420bd0ecc8643f1b62c4d0aa357ab15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:03.274538 25061 raft_consensus.cc:729] T a2b068a72a0d4b5bb40c456cee5c4a82 P 5420bd0ecc8643f1b62c4d0aa357ab15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5420bd0ecc8643f1b62c4d0aa357ab15, State: Initialized, Role: FOLLOWER
I20260620 18:38:03.275185 25061 consensus_queue.cc:260] T a2b068a72a0d4b5bb40c456cee5c4a82 P 5420bd0ecc8643f1b62c4d0aa357ab15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5420bd0ecc8643f1b62c4d0aa357ab15" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 37059 } } peers { permanent_uuid: "bb794b381e7242cb9eda42990aa23a55" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 35773 } } peers { permanent_uuid: "4dbd4eade5084e3cb512f2e9b2b9c87c" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 37675 } }
I20260620 18:38:03.276281 24909 heartbeater.cc:503] Master 127.23.115.62:46161 was elected leader, sending a full tablet report...
I20260620 18:38:03.276916 25061 ts_tablet_manager.cc:1428] T a2b068a72a0d4b5bb40c456cee5c4a82 P 5420bd0ecc8643f1b62c4d0aa357ab15: Time spent starting tablet: real 0.016s user 0.001s sys 0.012s
I20260620 18:38:03.321115 25068 raft_consensus.cc:482] T a2b068a72a0d4b5bb40c456cee5c4a82 P 5420bd0ecc8643f1b62c4d0aa357ab15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:38:03.321692 25068 raft_consensus.cc:504] T a2b068a72a0d4b5bb40c456cee5c4a82 P 5420bd0ecc8643f1b62c4d0aa357ab15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5420bd0ecc8643f1b62c4d0aa357ab15" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 37059 } } peers { permanent_uuid: "bb794b381e7242cb9eda42990aa23a55" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 35773 } } peers { permanent_uuid: "4dbd4eade5084e3cb512f2e9b2b9c87c" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 37675 } }
I20260620 18:38:03.323597 25068 leader_election.cc:288] T a2b068a72a0d4b5bb40c456cee5c4a82 P 5420bd0ecc8643f1b62c4d0aa357ab15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb794b381e7242cb9eda42990aa23a55 (127.23.115.1:35773), 4dbd4eade5084e3cb512f2e9b2b9c87c (127.23.115.3:37675)
I20260620 18:38:03.330539 25059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:03.324369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.2:59243 (local address 127.23.115.3:37675)
0620 18:38:03.324660 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:03.324668 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:03.325198 (+ 530us) server_negotiation.cc:407] Connection header received
0620 18:38:03.325384 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:03.325388 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:03.325464 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:03.325542 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:38:03.326073 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:03.327036 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:03.328845 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:03.329591 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:03.329758 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:03.329830 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:03.330185 (+ 355us) server_negotiation.cc:299] Negotiation successful
0620 18:38:03.330307 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260620 18:38:03.330554 25058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:03.324169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.2:49137 (local address 127.23.115.1:35773)
0620 18:38:03.324384 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:03.324391 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:03.325476 (+ 1085us) server_negotiation.cc:407] Connection header received
0620 18:38:03.325653 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:03.325657 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:03.325716 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:03.325786 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:38:03.326351 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:03.327114 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:03.328848 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:03.329600 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:03.329751 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:03.329827 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:03.330178 (+ 351us) server_negotiation.cc:299] Negotiation successful
0620 18:38:03.330300 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 18:38:03.330726 25069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:03.323980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.3:37675 (local address 127.23.115.2:59243)
0620 18:38:03.325064 (+ 1084us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:03.325081 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:03.325250 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:03.325570 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:03.325574 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:03.325599 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:38:03.325941 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:03.325953 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:03.327166 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:03.327170 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:03.328702 (+ 1532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:03.328714 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:03.329822 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:03.329829 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:03.329987 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:03.330058 (+ 71us) client_negotiation.cc:769] Sending connection context
0620 18:38:03.330147 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 18:38:03.330298 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":939,"mutex_wait_us":85,"thread_start_us":611,"threads_started":1}
I20260620 18:38:03.331048 25070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:03.324364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.1:35773 (local address 127.23.115.2:49137)
0620 18:38:03.325365 (+ 1001us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:03.325381 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:03.325507 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:03.325826 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:03.325831 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:03.325859 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:38:03.326210 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:03.326219 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:03.327244 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:03.327248 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:03.328710 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:03.328721 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:03.329855 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:03.329861 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:03.330022 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:03.330064 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 18:38:03.330139 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 18:38:03.330285 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":884,"mutex_wait_us":49,"thread_start_us":656,"threads_started":1}
I20260620 18:38:03.331715 24997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2b068a72a0d4b5bb40c456cee5c4a82" candidate_uuid: "5420bd0ecc8643f1b62c4d0aa357ab15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4dbd4eade5084e3cb512f2e9b2b9c87c" is_pre_election: true
I20260620 18:38:03.331872 24730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2b068a72a0d4b5bb40c456cee5c4a82" candidate_uuid: "5420bd0ecc8643f1b62c4d0aa357ab15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb794b381e7242cb9eda42990aa23a55" is_pre_election: true
I20260620 18:38:03.332204 24997 raft_consensus.cc:2454] T a2b068a72a0d4b5bb40c456cee5c4a82 P 4dbd4eade5084e3cb512f2e9b2b9c87c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5420bd0ecc8643f1b62c4d0aa357ab15 in term 0.
I20260620 18:38:03.332485 24730 raft_consensus.cc:2454] T a2b068a72a0d4b5bb40c456cee5c4a82 P bb794b381e7242cb9eda42990aa23a55 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5420bd0ecc8643f1b62c4d0aa357ab15 in term 0.
I20260620 18:38:03.332835 24799 leader_election.cc:302] T a2b068a72a0d4b5bb40c456cee5c4a82 P 5420bd0ecc8643f1b62c4d0aa357ab15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4dbd4eade5084e3cb512f2e9b2b9c87c, 5420bd0ecc8643f1b62c4d0aa357ab15; no voters:
I20260620 18:38:03.333209 25068 raft_consensus.cc:2781] T a2b068a72a0d4b5bb40c456cee5c4a82 P 5420bd0ecc8643f1b62c4d0aa357ab15 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:38:03.333356 25068 raft_consensus.cc:482] T a2b068a72a0d4b5bb40c456cee5c4a82 P 5420bd0ecc8643f1b62c4d0aa357ab15 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:38:03.333459 25068 raft_consensus.cc:3037] T a2b068a72a0d4b5bb40c456cee5c4a82 P 5420bd0ecc8643f1b62c4d0aa357ab15 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:03.335975 25068 raft_consensus.cc:504] T a2b068a72a0d4b5bb40c456cee5c4a82 P 5420bd0ecc8643f1b62c4d0aa357ab15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5420bd0ecc8643f1b62c4d0aa357ab15" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 37059 } } peers { permanent_uuid: "bb794b381e7242cb9eda42990aa23a55" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 35773 } } peers { permanent_uuid: "4dbd4eade5084e3cb512f2e9b2b9c87c" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 37675 } }
I20260620 18:38:03.336582 25068 leader_election.cc:288] T a2b068a72a0d4b5bb40c456cee5c4a82 P 5420bd0ecc8643f1b62c4d0aa357ab15 [CANDIDATE]: Term 1 election: Requested vote from peers bb794b381e7242cb9eda42990aa23a55 (127.23.115.1:35773), 4dbd4eade5084e3cb512f2e9b2b9c87c (127.23.115.3:37675)
I20260620 18:38:03.337008 24730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2b068a72a0d4b5bb40c456cee5c4a82" candidate_uuid: "5420bd0ecc8643f1b62c4d0aa357ab15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb794b381e7242cb9eda42990aa23a55"
I20260620 18:38:03.337191 24730 raft_consensus.cc:3037] T a2b068a72a0d4b5bb40c456cee5c4a82 P bb794b381e7242cb9eda42990aa23a55 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:03.337155 24997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2b068a72a0d4b5bb40c456cee5c4a82" candidate_uuid: "5420bd0ecc8643f1b62c4d0aa357ab15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4dbd4eade5084e3cb512f2e9b2b9c87c"
I20260620 18:38:03.337345 24997 raft_consensus.cc:3037] T a2b068a72a0d4b5bb40c456cee5c4a82 P 4dbd4eade5084e3cb512f2e9b2b9c87c [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:03.339406 24730 raft_consensus.cc:2454] T a2b068a72a0d4b5bb40c456cee5c4a82 P bb794b381e7242cb9eda42990aa23a55 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5420bd0ecc8643f1b62c4d0aa357ab15 in term 1.
I20260620 18:38:03.339911 24997 raft_consensus.cc:2454] T a2b068a72a0d4b5bb40c456cee5c4a82 P 4dbd4eade5084e3cb512f2e9b2b9c87c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5420bd0ecc8643f1b62c4d0aa357ab15 in term 1.
I20260620 18:38:03.339915 24798 leader_election.cc:302] T a2b068a72a0d4b5bb40c456cee5c4a82 P 5420bd0ecc8643f1b62c4d0aa357ab15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5420bd0ecc8643f1b62c4d0aa357ab15, bb794b381e7242cb9eda42990aa23a55; no voters:
I20260620 18:38:03.340238 25068 raft_consensus.cc:2781] T a2b068a72a0d4b5bb40c456cee5c4a82 P 5420bd0ecc8643f1b62c4d0aa357ab15 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:38:03.340642 25068 raft_consensus.cc:686] T a2b068a72a0d4b5bb40c456cee5c4a82 P 5420bd0ecc8643f1b62c4d0aa357ab15 [term 1 LEADER]: Becoming Leader. State: Replica: 5420bd0ecc8643f1b62c4d0aa357ab15, State: Running, Role: LEADER
I20260620 18:38:03.341173 25068 consensus_queue.cc:237] T a2b068a72a0d4b5bb40c456cee5c4a82 P 5420bd0ecc8643f1b62c4d0aa357ab15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5420bd0ecc8643f1b62c4d0aa357ab15" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 37059 } } peers { permanent_uuid: "bb794b381e7242cb9eda42990aa23a55" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 35773 } } peers { permanent_uuid: "4dbd4eade5084e3cb512f2e9b2b9c87c" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 37675 } }
I20260620 18:38:03.347028 24586 catalog_manager.cc:5495] T a2b068a72a0d4b5bb40c456cee5c4a82 P 5420bd0ecc8643f1b62c4d0aa357ab15 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5420bd0ecc8643f1b62c4d0aa357ab15 (127.23.115.2). New cstate: current_term: 1 leader_uuid: "5420bd0ecc8643f1b62c4d0aa357ab15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5420bd0ecc8643f1b62c4d0aa357ab15" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 37059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb794b381e7242cb9eda42990aa23a55" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 35773 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4dbd4eade5084e3cb512f2e9b2b9c87c" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 37675 } health_report { overall_health: UNKNOWN } } }
Jun 20 18:38:03 dist-test-slave-d645 krb5kdc[24541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980681, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.115.1@KRBTEST.COM
W20260620 18:38:03.379242 25043 tablet.cc:2369] T a2b068a72a0d4b5bb40c456cee5c4a82 P 4dbd4eade5084e3cb512f2e9b2b9c87c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:38:03.383127 25058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:03.367372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48862 (local address 127.23.115.1:35773)
0620 18:38:03.367566 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:03.367572 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:03.368042 (+ 470us) server_negotiation.cc:407] Connection header received
0620 18:38:03.368209 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:03.368216 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:03.368286 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:03.368372 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:03.369812 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:03.370563 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:03.371800 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:03.372210 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:03.372318 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:03.377170 (+ 4852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:03.377195 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:03.377212 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:03.377268 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:03.380193 (+ 2925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:03.380887 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:03.380895 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:03.380898 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:03.381130 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:03.381537 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:03.381542 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:03.381544 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:03.381914 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:03.382183 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:03.382504 (+ 321us) server_negotiation.cc:299] Negotiation successful
0620 18:38:03.382757 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260620 18:38:03.396919 24910 tablet.cc:2369] T a2b068a72a0d4b5bb40c456cee5c4a82 P 5420bd0ecc8643f1b62c4d0aa357ab15: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 18:38:03 dist-test-slave-d645 krb5kdc[24541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980683, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 18:38:03 dist-test-slave-d645 krb5kdc[24541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980683, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.115.1@KRBTEST.COM
I20260620 18:38:03.428009 25058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:03.413832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48864 (local address 127.23.115.1:35773)
0620 18:38:03.414053 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:03.414059 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:03.414335 (+ 276us) server_negotiation.cc:407] Connection header received
0620 18:38:03.414516 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:03.414521 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:03.414591 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:03.414683 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:03.416346 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:03.417155 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:03.418578 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:03.419082 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:03.419189 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:03.423005 (+ 3816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:03.423010 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:03.423014 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:03.423049 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:03.425533 (+ 2484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:03.426179 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:03.426183 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:03.426186 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:03.426315 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:03.426716 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:03.426721 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:03.426723 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:03.427043 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:03.427246 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:03.427517 (+ 271us) server_negotiation.cc:299] Negotiation successful
0620 18:38:03.427695 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260620 18:38:03.434644 24776 tablet.cc:2369] T a2b068a72a0d4b5bb40c456cee5c4a82 P bb794b381e7242cb9eda42990aa23a55: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 18:38:03 dist-test-slave-d645 krb5kdc[24541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980683, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 18:38:03 dist-test-slave-d645 krb5kdc[24541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980683, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.115.1@KRBTEST.COM
I20260620 18:38:03.470678 25058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:03.456500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48878 (local address 127.23.115.1:35773)
0620 18:38:03.456815 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:03.456823 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:03.457029 (+ 206us) server_negotiation.cc:407] Connection header received
0620 18:38:03.457284 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:03.457289 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:03.457375 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:03.457468 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:03.459230 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:03.460002 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:03.461220 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:03.461690 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:03.461807 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:03.465761 (+ 3954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:03.465768 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:03.465771 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:03.465811 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:03.468202 (+ 2391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:03.468815 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:03.468819 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:03.468821 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:03.468944 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:03.469299 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:03.469303 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:03.469306 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:03.469593 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:03.469829 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:03.470096 (+ 267us) server_negotiation.cc:299] Negotiation successful
0620 18:38:03.470343 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
W20260620 18:38:03.472388 24690 tablet_service.cc:2908] Rejecting scan request for tablet a2b068a72a0d4b5bb40c456cee5c4a82: Uninitialized: clean time has not yet been initialized
I20260620 18:38:03.475445 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 24647
I20260620 18:38:03.487069 25058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:03.476334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48888 (local address 127.23.115.1:35773)
0620 18:38:03.476504 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:03.476510 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:03.476792 (+ 282us) server_negotiation.cc:407] Connection header received
0620 18:38:03.476944 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:03.476950 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:03.477027 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:03.477107 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:03.478182 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:03.478853 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:03.479990 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:03.480401 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:03.480515 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:03.482476 (+ 1961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:03.482481 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:03.482484 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:03.482512 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:03.484793 (+ 2281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:03.485375 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:03.485380 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:03.485383 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:03.485492 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:03.485804 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:03.485809 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:03.485813 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:03.486048 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:03.486299 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:03.486549 (+ 250us) server_negotiation.cc:299] Negotiation successful
0620 18:38:03.486769 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 18:38:03.487715 24770 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:38:03.576776 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 24647
I20260620 18:38:03.589759 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 24781
Jun 20 18:38:03 dist-test-slave-d645 krb5kdc[24541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980683, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.115.2@KRBTEST.COM
I20260620 18:38:03.604935 25055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:03.590523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50586 (local address 127.23.115.2:37059)
0620 18:38:03.590732 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:03.590738 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:03.590757 (+ 19us) server_negotiation.cc:407] Connection header received
0620 18:38:03.590978 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:03.590984 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:03.591051 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:03.591173 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:03.592257 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:03.592965 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:03.594183 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:03.594681 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:03.594811 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:03.599316 (+ 4505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:03.599332 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:03.599348 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:03.599387 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:03.602048 (+ 2661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:03.602722 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:03.602728 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:03.602730 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:03.602866 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:03.603248 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:03.603251 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:03.603254 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:03.603789 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:03.604074 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:03.604378 (+ 304us) server_negotiation.cc:299] Negotiation successful
0620 18:38:03.604623 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 18:38:03.605856 24904 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:38:03.700325 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 24781
I20260620 18:38:03.713881 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 24914
Jun 20 18:38:03 dist-test-slave-d645 krb5kdc[24541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980683, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.115.3@KRBTEST.COM
I20260620 18:38:03.728482 25059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:03.714646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57620 (local address 127.23.115.3:37675)
0620 18:38:03.714844 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:03.714850 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:03.714869 (+ 19us) server_negotiation.cc:407] Connection header received
0620 18:38:03.715051 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:03.715055 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:03.715129 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:03.715222 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:03.716436 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:03.717160 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:03.718433 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:03.718844 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:03.719045 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:03.722906 (+ 3861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:03.722920 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:03.722962 (+ 42us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:03.723002 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:03.725922 (+ 2920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:03.726542 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:03.726551 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:03.726555 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:03.726688 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:03.727058 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:03.727063 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:03.727067 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:03.727402 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:03.727661 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:03.727971 (+ 310us) server_negotiation.cc:299] Negotiation successful
0620 18:38:03.728176 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 18:38:03.729044 25037 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:38:03.815878 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 24914
I20260620 18:38:03.828614 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 24554
I20260620 18:38:03.829746 24616 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:38:03.940245 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 24554
2026-06-20T18:38:03Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (3943 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980676253068-24012-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:38:04 dist-test-slave-d645 krb5kdc[25090](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:38:04 dist-test-slave-d645 krb5kdc[25090](info): set up 2 sockets
Jun 20 18:38:04 dist-test-slave-d645 krb5kdc[25090](info): commencing operation
krb5kdc: starting...
W20260620 18:38:04.943161 24012 mini_kdc.cc:121] Time spent starting KDC: real 0.965s user 0.005s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980676253068-24012-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980676253068-24012-0/krb5kdc/test-user.keytab.
Jun 20 18:38:05 dist-test-slave-d645 krb5kdc[25090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980685, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:38:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:38:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.115.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.115.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.115.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab.
Entry for principal kudu/127.23.115.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab.
I20260620 18:38:05.117941 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980676253068-24012-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980676253068-24012-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980676253068-24012-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.115.62:40053
--webserver_interface=127.23.115.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab
--principal=kudu/127.23.115.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:43843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.115.62:40053
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:38:05.423362 25103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:05.423674 25103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:05.423810 25103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:05.431392 25103 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:38:05.431512 25103 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 18:38:05.431579 25103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:05.431663 25103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:05.431740 25103 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:38:05.431820 25103 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:38:05.441912 25103 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:43843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980676253068-24012-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980676253068-24012-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.115.62:40053
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab
--principal=kudu/127.23.115.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.115.62:40053
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.115.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980676253068-24012-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:38:05.443254 25103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:05.444507 25103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:05.455497 25108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:05.455420 25109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:05.455454 25111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:05.456750 25103 server_base.cc:1034] running on GCE node
I20260620 18:38:05.457522 25103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:05.458902 25103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:05.460170 25103 hybrid_clock.cc:648] HybridClock initialized: now 1781980685460106 us; error 33 us; skew 500 ppm
I20260620 18:38:05.460497 25103 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:38:05 dist-test-slave-d645 krb5kdc[25090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980685, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:05.466540 25103 init.cc:377] Logged in from keytab as kudu/127.23.115.62@KRBTEST.COM (short username kudu)
I20260620 18:38:05.468099 25103 webserver.cc:460] Webserver started at http://127.23.115.62:42069/ using document root <none> and password file <none>
I20260620 18:38:05.468794 25103 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:05.468894 25103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:05.469192 25103 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:38:05.472095 25103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980676253068-24012-0/minicluster-data/master-0/data/instance:
uuid: "d9e48bfe423541ea965d5203d2ce8dbe"
format_stamp: "Formatted at 2026-06-20 18:38:05 on dist-test-slave-d645"
server_key: "97968ac2785c4340152ca7987da17391"
server_key_iv: "8eafb04a63eac6bf3a7706ac5dadc295"
server_key_version: "encryptionkey@0"
I20260620 18:38:05.472826 25103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980676253068-24012-0/minicluster-data/master-0/wal/instance:
uuid: "d9e48bfe423541ea965d5203d2ce8dbe"
format_stamp: "Formatted at 2026-06-20 18:38:05 on dist-test-slave-d645"
server_key: "97968ac2785c4340152ca7987da17391"
server_key_iv: "8eafb04a63eac6bf3a7706ac5dadc295"
server_key_version: "encryptionkey@0"
I20260620 18:38:05.477854 25103 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260620 18:38:05.481199 25118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:05.482533 25103 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 18:38:05.482724 25103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980676253068-24012-0/minicluster-data/master-0/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980676253068-24012-0/minicluster-data/master-0/wal
uuid: "d9e48bfe423541ea965d5203d2ce8dbe"
format_stamp: "Formatted at 2026-06-20 18:38:05 on dist-test-slave-d645"
server_key: "97968ac2785c4340152ca7987da17391"
server_key_iv: "8eafb04a63eac6bf3a7706ac5dadc295"
server_key_version: "encryptionkey@0"
I20260620 18:38:05.482920 25103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980676253068-24012-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980676253068-24012-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980676253068-24012-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:05.504827 25103 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:05.509083 25103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:05.509361 25103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:05.531287 25103 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.62:40053
I20260620 18:38:05.531260 25179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.62:40053 every 8 connection(s)
I20260620 18:38:05.532610 25103 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
I20260620 18:38:05.534300 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 25103
I20260620 18:38:05.534524 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980676253068-24012-0/minicluster-data/master-0/wal/instance
I20260620 18:38:05.535348 24012 external_mini_cluster.cc:1428] Setting key bdbca0e85276696a3f068db2578b59bb
I20260620 18:38:05.537415 25180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 18:38:05 dist-test-slave-d645 krb5kdc[25090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980685, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.115.62@KRBTEST.COM
I20260620 18:38:05.551208 25180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9e48bfe423541ea965d5203d2ce8dbe: Bootstrap starting.
I20260620 18:38:05.555158 25180 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d9e48bfe423541ea965d5203d2ce8dbe: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:05.556330 25180 log.cc:824] T 00000000000000000000000000000000 P d9e48bfe423541ea965d5203d2ce8dbe: Log is configured to *not* fsync() on all Append() calls
I20260620 18:38:05.559707 25180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9e48bfe423541ea965d5203d2ce8dbe: No bootstrap required, opened a new log
I20260620 18:38:05.559738 25183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:05.537445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56554 (local address 127.23.115.62:40053)
0620 18:38:05.538310 (+ 865us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:05.538321 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:05.538362 (+ 41us) server_negotiation.cc:407] Connection header received
0620 18:38:05.540022 (+ 1660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:05.540044 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:05.540422 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:05.540746 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:05.542588 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:05.544027 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:05.545758 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:05.546760 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:05.546889 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:05.551588 (+ 4699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:05.551599 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:05.551619 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:05.551668 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:05.555293 (+ 3625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:05.555970 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:05.555979 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:05.555988 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:05.556198 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:05.556596 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:05.556601 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:05.556603 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:05.557256 (+ 653us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:05.557463 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:05.558008 (+ 545us) server_negotiation.cc:299] Negotiation successful
0620 18:38:05.558449 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":659,"thread_start_us":437,"threads_started":1}
I20260620 18:38:05.567559 25180 raft_consensus.cc:348] T 00000000000000000000000000000000 P d9e48bfe423541ea965d5203d2ce8dbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9e48bfe423541ea965d5203d2ce8dbe" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 40053 } }
I20260620 18:38:05.567905 25180 raft_consensus.cc:374] T 00000000000000000000000000000000 P d9e48bfe423541ea965d5203d2ce8dbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:05.567977 25180 raft_consensus.cc:729] T 00000000000000000000000000000000 P d9e48bfe423541ea965d5203d2ce8dbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9e48bfe423541ea965d5203d2ce8dbe, State: Initialized, Role: FOLLOWER
I20260620 18:38:05.568547 25180 consensus_queue.cc:260] T 00000000000000000000000000000000 P d9e48bfe423541ea965d5203d2ce8dbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9e48bfe423541ea965d5203d2ce8dbe" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 40053 } }
I20260620 18:38:05.568771 25180 raft_consensus.cc:388] T 00000000000000000000000000000000 P d9e48bfe423541ea965d5203d2ce8dbe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:38:05.568930 25180 raft_consensus.cc:482] T 00000000000000000000000000000000 P d9e48bfe423541ea965d5203d2ce8dbe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:38:05.569082 25180 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d9e48bfe423541ea965d5203d2ce8dbe [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:05.571161 25180 raft_consensus.cc:504] T 00000000000000000000000000000000 P d9e48bfe423541ea965d5203d2ce8dbe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9e48bfe423541ea965d5203d2ce8dbe" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 40053 } }
I20260620 18:38:05.571602 25180 leader_election.cc:302] T 00000000000000000000000000000000 P d9e48bfe423541ea965d5203d2ce8dbe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d9e48bfe423541ea965d5203d2ce8dbe; no voters:
I20260620 18:38:05.572103 25180 leader_election.cc:288] T 00000000000000000000000000000000 P d9e48bfe423541ea965d5203d2ce8dbe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:38:05.572255 25185 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d9e48bfe423541ea965d5203d2ce8dbe [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:38:05.572868 25185 raft_consensus.cc:686] T 00000000000000000000000000000000 P d9e48bfe423541ea965d5203d2ce8dbe [term 1 LEADER]: Becoming Leader. State: Replica: d9e48bfe423541ea965d5203d2ce8dbe, State: Running, Role: LEADER
I20260620 18:38:05.573385 25185 consensus_queue.cc:237] T 00000000000000000000000000000000 P d9e48bfe423541ea965d5203d2ce8dbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9e48bfe423541ea965d5203d2ce8dbe" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 40053 } }
I20260620 18:38:05.574082 25180 sys_catalog.cc:564] T 00000000000000000000000000000000 P d9e48bfe423541ea965d5203d2ce8dbe [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:38:05.576172 25187 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9e48bfe423541ea965d5203d2ce8dbe [sys.catalog]: SysCatalogTable state changed. Reason: New leader d9e48bfe423541ea965d5203d2ce8dbe. Latest consensus state: current_term: 1 leader_uuid: "d9e48bfe423541ea965d5203d2ce8dbe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9e48bfe423541ea965d5203d2ce8dbe" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 40053 } } }
I20260620 18:38:05.576542 25187 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9e48bfe423541ea965d5203d2ce8dbe [sys.catalog]: This master's current role is: LEADER
I20260620 18:38:05.576987 25186 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9e48bfe423541ea965d5203d2ce8dbe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d9e48bfe423541ea965d5203d2ce8dbe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9e48bfe423541ea965d5203d2ce8dbe" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 40053 } } }
I20260620 18:38:05.577785 25186 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9e48bfe423541ea965d5203d2ce8dbe [sys.catalog]: This master's current role is: LEADER
I20260620 18:38:05.578794 25193 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:38:05.584299 25193 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:38:05.593329 25193 catalog_manager.cc:1347] Generated new cluster ID: f3b0875454544b298809f210ff2d58bc
I20260620 18:38:05.593454 25193 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:38:05.616189 25193 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:38:05.618080 25193 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:38:05.629330 25193 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d9e48bfe423541ea965d5203d2ce8dbe: Generated new TSK 0
I20260620 18:38:05.630415 25193 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.115.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.115.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.115.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.1.keytab.
Entry for principal kudu/127.23.115.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.1.keytab.
I20260620 18:38:05.670655 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980676253068-24012-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980676253068-24012-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980676253068-24012-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980676253068-24012-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.115.1:0
--local_ip_for_outbound_sockets=127.23.115.1
--webserver_interface=127.23.115.1
--webserver_port=0
--tserver_master_addrs=127.23.115.62:40053
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.1.keytab
--principal=kudu/127.23.115.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:43843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:38:05.984459 25206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:05.984725 25206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:05.984834 25206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:05.992547 25206 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:05.992655 25206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:05.992758 25206 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 18:38:05.992837 25206 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.115.1
I20260620 18:38:06.002537 25206 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:43843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980676253068-24012-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980676253068-24012-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.1.keytab
--principal=kudu/127.23.115.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.115.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980676253068-24012-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.115.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.115.62:40053
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.115.1
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980676253068-24012-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:38:06.003762 25206 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:06.005096 25206 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:06.020331 25211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:06.020395 25214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:06.020399 25212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:06.021070 25206 server_base.cc:1034] running on GCE node
I20260620 18:38:06.021705 25206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:06.022687 25206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:06.023914 25206 hybrid_clock.cc:648] HybridClock initialized: now 1781980686023824 us; error 64 us; skew 500 ppm
I20260620 18:38:06.024216 25206 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:38:06 dist-test-slave-d645 krb5kdc[25090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980686, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:06.028023 25206 init.cc:377] Logged in from keytab as kudu/127.23.115.1@KRBTEST.COM (short username kudu)
I20260620 18:38:06.029521 25206 webserver.cc:460] Webserver started at http://127.23.115.1:45389/ using document root <none> and password file <none>
I20260620 18:38:06.030211 25206 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:06.030328 25206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:06.030634 25206 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:38:06.033480 25206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980676253068-24012-0/minicluster-data/ts-0/data/instance:
uuid: "26f1de5d3e9744eebd1deb54b454f585"
format_stamp: "Formatted at 2026-06-20 18:38:06 on dist-test-slave-d645"
server_key: "b4d2fe80b38f77c617aac2541f3711a0"
server_key_iv: "4409de2f4b5fb0744e181bd556e1c54e"
server_key_version: "encryptionkey@0"
I20260620 18:38:06.034240 25206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980676253068-24012-0/minicluster-data/ts-0/wal/instance:
uuid: "26f1de5d3e9744eebd1deb54b454f585"
format_stamp: "Formatted at 2026-06-20 18:38:06 on dist-test-slave-d645"
server_key: "b4d2fe80b38f77c617aac2541f3711a0"
server_key_iv: "4409de2f4b5fb0744e181bd556e1c54e"
server_key_version: "encryptionkey@0"
I20260620 18:38:06.039208 25206 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 18:38:06.042625 25221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:06.044029 25206 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:38:06.044196 25206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980676253068-24012-0/minicluster-data/ts-0/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980676253068-24012-0/minicluster-data/ts-0/wal
uuid: "26f1de5d3e9744eebd1deb54b454f585"
format_stamp: "Formatted at 2026-06-20 18:38:06 on dist-test-slave-d645"
server_key: "b4d2fe80b38f77c617aac2541f3711a0"
server_key_iv: "4409de2f4b5fb0744e181bd556e1c54e"
server_key_version: "encryptionkey@0"
I20260620 18:38:06.044392 25206 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980676253068-24012-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980676253068-24012-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980676253068-24012-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:06.059032 25206 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:06.063208 25206 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:06.063452 25206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:06.066347 25206 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:38:06.066439 25206 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:06.066582 25206 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:38:06.066635 25206 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 18:38:06 dist-test-slave-d645 krb5kdc[25090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980686, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.1@KRBTEST.COM for kudu/127.23.115.62@KRBTEST.COM
I20260620 18:38:06.092186 25230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:06.067534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.1:38959 (local address 127.23.115.62:40053)
0620 18:38:06.067999 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:06.068005 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:06.069327 (+ 1322us) server_negotiation.cc:407] Connection header received
0620 18:38:06.071779 (+ 2452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:06.071784 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:06.071871 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:06.071966 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:06.075150 (+ 3184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:06.076099 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:06.078583 (+ 2484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:06.079078 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:06.079193 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:06.084485 (+ 5292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:06.084492 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:06.084497 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:06.084543 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:06.087707 (+ 3164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:06.089030 (+ 1323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:06.089036 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:06.089039 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:06.089236 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:06.089805 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:06.089809 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:06.089811 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:06.090174 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:06.090350 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:06.091384 (+ 1034us) server_negotiation.cc:299] Negotiation successful
0620 18:38:06.091806 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":225,"threads_started":1}
I20260620 18:38:06.093794 25231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:06.068128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:40053 (local address 127.23.115.1:38959)
0620 18:38:06.069144 (+ 1016us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:06.069200 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:06.071455 (+ 2255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:06.072527 (+ 1072us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:06.072540 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:06.073939 (+ 1399us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:06.074893 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:06.074924 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:06.076290 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:06.076296 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:06.078420 (+ 2124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:06.078436 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:06.079333 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:06.079345 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:06.079629 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:06.080444 (+ 815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:06.080483 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:06.084193 (+ 3710us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:06.087863 (+ 3670us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:06.087872 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:06.087891 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:06.088746 (+ 855us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:06.089376 (+ 630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:06.089382 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:06.089387 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:06.089664 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:06.090333 (+ 669us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:06.090342 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:06.090872 (+ 530us) client_negotiation.cc:769] Sending connection context
0620 18:38:06.091267 (+ 395us) client_negotiation.cc:240] Negotiation successful
0620 18:38:06.092112 (+ 845us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":736,"thread_start_us":501,"threads_started":1}
I20260620 18:38:06.110149 25206 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.1:41579
I20260620 18:38:06.110184 25341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.1:41579 every 8 connection(s)
I20260620 18:38:06.111439 25206 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980676253068-24012-0/minicluster-data/ts-0/data/info.pb
I20260620 18:38:06.113447 25342 heartbeater.cc:347] Connected to a master server at 127.23.115.62:40053
I20260620 18:38:06.113575 25230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:06.100625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.1:51491 (local address 127.23.115.62:40053)
0620 18:38:06.100813 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:06.100820 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:06.100868 (+ 48us) server_negotiation.cc:407] Connection header received
0620 18:38:06.101026 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:06.101032 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:06.101130 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:06.101263 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:06.102503 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:06.103526 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:06.105361 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:06.105805 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:06.105916 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:06.108016 (+ 2100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:06.108022 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:06.108025 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:06.108059 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:06.110782 (+ 2723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:06.111486 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:06.111492 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:06.111496 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:06.111651 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:06.111986 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:06.111991 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:06.111995 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:06.112356 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:06.112500 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:06.113028 (+ 528us) server_negotiation.cc:299] Negotiation successful
0620 18:38:06.113248 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 18:38:06.113692 25317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:06.100172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:40053 (local address 127.23.115.1:51491)
0620 18:38:06.100776 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:06.100795 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:06.100899 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:06.101312 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:06.101317 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:06.101915 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:06.102366 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:06.102377 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:06.103699 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:06.103702 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:06.105218 (+ 1516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:06.105234 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:06.105983 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:06.105989 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:06.106151 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:06.106734 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:06.106751 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:06.107774 (+ 1023us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:06.110909 (+ 3135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:06.110914 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:06.110919 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:06.111378 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:06.111749 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:06.111754 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:06.111758 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:06.111890 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:06.112562 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:06.112567 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:06.112778 (+ 211us) client_negotiation.cc:769] Sending connection context
0620 18:38:06.112898 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 18:38:06.113207 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":308,"threads_started":1}
I20260620 18:38:06.113771 25342 heartbeater.cc:464] Registering TS with master...
I20260620 18:38:06.114527 25342 heartbeater.cc:511] Master 127.23.115.62:40053 requested a full tablet report, sending...
I20260620 18:38:06.116564 25135 ts_manager.cc:194] Registered new tserver with Master: 26f1de5d3e9744eebd1deb54b454f585 (127.23.115.1:41579)
I20260620 18:38:06.116694 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 25206
I20260620 18:38:06.116906 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980676253068-24012-0/minicluster-data/ts-0/wal/instance
I20260620 18:38:06.117594 24012 external_mini_cluster.cc:1428] Setting key 9ef8d4aa99a55dec3d80e87e351d3b8a
I20260620 18:38:06.118084 25135 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.115.1@KRBTEST.COM'} at 127.23.115.1:38959
WARNING: no policy specified for kudu/127.23.115.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.115.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.115.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.2.keytab.
Entry for principal kudu/127.23.115.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.2.keytab.
I20260620 18:38:06.154320 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980676253068-24012-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980676253068-24012-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980676253068-24012-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980676253068-24012-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.115.2:0
--local_ip_for_outbound_sockets=127.23.115.2
--webserver_interface=127.23.115.2
--webserver_port=0
--tserver_master_addrs=127.23.115.62:40053
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.2.keytab
--principal=kudu/127.23.115.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:43843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:38:06.455649 25346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:06.455924 25346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:06.456058 25346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:06.463135 25346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:06.463279 25346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:06.463397 25346 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 18:38:06.463505 25346 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.115.2
I20260620 18:38:06.473101 25346 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:43843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980676253068-24012-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980676253068-24012-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.2.keytab
--principal=kudu/127.23.115.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.115.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980676253068-24012-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.115.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.115.62:40053
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.115.2
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980676253068-24012-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:38:06.474370 25346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:06.475670 25346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:06.488113 25351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:06.488365 25354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:06.488374 25352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:06.489468 25346 server_base.cc:1034] running on GCE node
I20260620 18:38:06.490088 25346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:06.490978 25346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:06.492239 25346 hybrid_clock.cc:648] HybridClock initialized: now 1781980686492159 us; error 47 us; skew 500 ppm
I20260620 18:38:06.492555 25346 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:38:06 dist-test-slave-d645 krb5kdc[25090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980686, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:06.495523 25346 init.cc:377] Logged in from keytab as kudu/127.23.115.2@KRBTEST.COM (short username kudu)
I20260620 18:38:06.496883 25346 webserver.cc:460] Webserver started at http://127.23.115.2:41067/ using document root <none> and password file <none>
I20260620 18:38:06.497566 25346 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:06.497656 25346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:06.497920 25346 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:38:06.500685 25346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980676253068-24012-0/minicluster-data/ts-1/data/instance:
uuid: "afde26b99c8d4e2e8ac019f9fa3fb40c"
format_stamp: "Formatted at 2026-06-20 18:38:06 on dist-test-slave-d645"
server_key: "15dbcc605b7ae2721ada99c528105a0f"
server_key_iv: "7fda38c50f45046a95ea5d8ab6c72966"
server_key_version: "encryptionkey@0"
I20260620 18:38:06.501405 25346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980676253068-24012-0/minicluster-data/ts-1/wal/instance:
uuid: "afde26b99c8d4e2e8ac019f9fa3fb40c"
format_stamp: "Formatted at 2026-06-20 18:38:06 on dist-test-slave-d645"
server_key: "15dbcc605b7ae2721ada99c528105a0f"
server_key_iv: "7fda38c50f45046a95ea5d8ab6c72966"
server_key_version: "encryptionkey@0"
I20260620 18:38:06.506444 25346 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 18:38:06.509938 25361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:06.511453 25346 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 18:38:06.511616 25346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980676253068-24012-0/minicluster-data/ts-1/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980676253068-24012-0/minicluster-data/ts-1/wal
uuid: "afde26b99c8d4e2e8ac019f9fa3fb40c"
format_stamp: "Formatted at 2026-06-20 18:38:06 on dist-test-slave-d645"
server_key: "15dbcc605b7ae2721ada99c528105a0f"
server_key_iv: "7fda38c50f45046a95ea5d8ab6c72966"
server_key_version: "encryptionkey@0"
I20260620 18:38:06.511861 25346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980676253068-24012-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980676253068-24012-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980676253068-24012-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:06.522289 25346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:06.526358 25346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:06.526608 25346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:06.529289 25346 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:38:06.529381 25346 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:06.529511 25346 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:38:06.529565 25346 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 18:38:06 dist-test-slave-d645 krb5kdc[25090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980686, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.2@KRBTEST.COM for kudu/127.23.115.62@KRBTEST.COM
I20260620 18:38:06.552914 25230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:06.529930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.2:37685 (local address 127.23.115.62:40053)
0620 18:38:06.530115 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:06.530121 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:06.531476 (+ 1355us) server_negotiation.cc:407] Connection header received
0620 18:38:06.533695 (+ 2219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:06.533701 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:06.533788 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:06.533886 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:06.536117 (+ 2231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:06.537244 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:06.540071 (+ 2827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:06.540575 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:06.540696 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:06.545010 (+ 4314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:06.545018 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:06.545023 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:06.545067 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:06.548794 (+ 3727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:06.549909 (+ 1115us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:06.549915 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:06.549919 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:06.550069 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:06.550671 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:06.550675 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:06.550679 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:06.551115 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:06.551278 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:06.552085 (+ 807us) server_negotiation.cc:299] Negotiation successful
0620 18:38:06.552325 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 18:38:06.554240 25370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:06.530484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:40053 (local address 127.23.115.2:37685)
0620 18:38:06.531314 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:06.531360 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:06.533329 (+ 1969us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:06.534150 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:06.534162 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:06.534983 (+ 821us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:06.535878 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:06.535910 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:06.537460 (+ 1550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:06.537465 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:06.539869 (+ 2404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:06.539885 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:06.540905 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:06.540912 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:06.541097 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:06.541791 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:06.541821 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:06.544700 (+ 2879us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:06.548985 (+ 4285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:06.548995 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:06.549018 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:06.549747 (+ 729us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:06.550212 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:06.550217 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:06.550221 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:06.550499 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:06.551269 (+ 770us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:06.551277 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:06.551677 (+ 400us) client_negotiation.cc:769] Sending connection context
0620 18:38:06.551966 (+ 289us) client_negotiation.cc:240] Negotiation successful
0620 18:38:06.552867 (+ 901us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":370,"threads_started":1}
I20260620 18:38:06.574005 25230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:06.560153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.2:44369 (local address 127.23.115.62:40053)
0620 18:38:06.560298 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:06.560302 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:06.561258 (+ 956us) server_negotiation.cc:407] Connection header received
0620 18:38:06.561455 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:06.561459 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:06.561520 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:06.561621 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:06.562652 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:06.563625 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:06.565406 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:06.565811 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:06.565915 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:06.568542 (+ 2627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:06.568548 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:06.568551 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:06.568583 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:06.571103 (+ 2520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:06.571911 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:06.571915 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:06.571918 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:06.572081 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:06.572547 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:06.572551 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:06.572553 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:06.572924 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:06.573051 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:06.573529 (+ 478us) server_negotiation.cc:299] Negotiation successful
0620 18:38:06.573734 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 18:38:06.574301 25434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:06.560403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:40053 (local address 127.23.115.2:44369)
0620 18:38:06.561151 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:06.561174 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:06.561322 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:06.561700 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:06.561704 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:06.562177 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:06.562519 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:06.562528 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:06.563785 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:06.563791 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:06.565274 (+ 1483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:06.565290 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:06.566002 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:06.566011 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:06.566163 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:06.567120 (+ 957us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:06.567149 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:06.568362 (+ 1213us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:06.571225 (+ 2863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:06.571234 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:06.571239 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:06.571758 (+ 519us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:06.572208 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:06.572216 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:06.572219 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:06.572406 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:06.573040 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:06.573045 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:06.573264 (+ 219us) client_negotiation.cc:769] Sending connection context
0620 18:38:06.573416 (+ 152us) client_negotiation.cc:240] Negotiation successful
0620 18:38:06.573793 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":412,"threads_started":1}
I20260620 18:38:06.583449 25346 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.2:35569
I20260620 18:38:06.583467 25480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.2:35569 every 8 connection(s)
I20260620 18:38:06.584621 25346 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980676253068-24012-0/minicluster-data/ts-1/data/info.pb
I20260620 18:38:06.586813 25481 heartbeater.cc:347] Connected to a master server at 127.23.115.62:40053
I20260620 18:38:06.587056 25481 heartbeater.cc:464] Registering TS with master...
I20260620 18:38:06.587721 25481 heartbeater.cc:511] Master 127.23.115.62:40053 requested a full tablet report, sending...
I20260620 18:38:06.589057 25134 ts_manager.cc:194] Registered new tserver with Master: afde26b99c8d4e2e8ac019f9fa3fb40c (127.23.115.2:35569)
I20260620 18:38:06.590076 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 25346
I20260620 18:38:06.590083 25134 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.115.2@KRBTEST.COM'} at 127.23.115.2:37685
I20260620 18:38:06.590320 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980676253068-24012-0/minicluster-data/ts-1/wal/instance
I20260620 18:38:06.590772 24012 external_mini_cluster.cc:1428] Setting key 3ff1e64a7150c85830f0b3ef023a7025
WARNING: no policy specified for kudu/127.23.115.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.115.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.115.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.3.keytab.
Entry for principal kudu/127.23.115.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.3.keytab.
I20260620 18:38:06.631049 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980676253068-24012-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980676253068-24012-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980676253068-24012-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980676253068-24012-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.115.3:0
--local_ip_for_outbound_sockets=127.23.115.3
--webserver_interface=127.23.115.3
--webserver_port=0
--tserver_master_addrs=127.23.115.62:40053
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.3.keytab
--principal=kudu/127.23.115.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:43843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:38:06.943876 25485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:06.944149 25485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:06.944245 25485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:06.951584 25485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:06.951696 25485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:06.951790 25485 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 18:38:06.951867 25485 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.115.3
I20260620 18:38:06.961633 25485 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:43843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980676253068-24012-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980676253068-24012-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.3.keytab
--principal=kudu/127.23.115.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.115.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980676253068-24012-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.115.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.115.62:40053
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.115.3
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980676253068-24012-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:38:06.962801 25485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:06.964057 25485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:06.977097 25491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:06.977080 25490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:06.977366 25493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:06.977900 25485 server_base.cc:1034] running on GCE node
I20260620 18:38:06.978543 25485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:06.979601 25485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:06.980901 25485 hybrid_clock.cc:648] HybridClock initialized: now 1781980686980818 us; error 74 us; skew 500 ppm
I20260620 18:38:06.981220 25485 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:38:06 dist-test-slave-d645 krb5kdc[25090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980686, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:06.984222 25485 init.cc:377] Logged in from keytab as kudu/127.23.115.3@KRBTEST.COM (short username kudu)
I20260620 18:38:06.985564 25485 webserver.cc:460] Webserver started at http://127.23.115.3:44993/ using document root <none> and password file <none>
I20260620 18:38:06.986215 25485 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:06.986317 25485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:06.986601 25485 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:38:06.989401 25485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980676253068-24012-0/minicluster-data/ts-2/data/instance:
uuid: "39e676f08fe246fa9d7590ab40edd09e"
format_stamp: "Formatted at 2026-06-20 18:38:06 on dist-test-slave-d645"
server_key: "31790e599e46b137437fff16fb68be94"
server_key_iv: "8ed8c901bd36aef40e5157ffdbc7254e"
server_key_version: "encryptionkey@0"
I20260620 18:38:06.990106 25485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980676253068-24012-0/minicluster-data/ts-2/wal/instance:
uuid: "39e676f08fe246fa9d7590ab40edd09e"
format_stamp: "Formatted at 2026-06-20 18:38:06 on dist-test-slave-d645"
server_key: "31790e599e46b137437fff16fb68be94"
server_key_iv: "8ed8c901bd36aef40e5157ffdbc7254e"
server_key_version: "encryptionkey@0"
I20260620 18:38:06.995043 25485 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260620 18:38:06.998562 25500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:06.999933 25485 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:38:07.000073 25485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980676253068-24012-0/minicluster-data/ts-2/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980676253068-24012-0/minicluster-data/ts-2/wal
uuid: "39e676f08fe246fa9d7590ab40edd09e"
format_stamp: "Formatted at 2026-06-20 18:38:06 on dist-test-slave-d645"
server_key: "31790e599e46b137437fff16fb68be94"
server_key_iv: "8ed8c901bd36aef40e5157ffdbc7254e"
server_key_version: "encryptionkey@0"
I20260620 18:38:07.000267 25485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980676253068-24012-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980676253068-24012-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980676253068-24012-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:07.043187 25485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:07.047143 25485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:07.047384 25485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:07.049976 25485 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:38:07.050050 25485 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:07.050205 25485 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:38:07.050263 25485 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 18:38:07 dist-test-slave-d645 krb5kdc[25090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980686, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.3@KRBTEST.COM for kudu/127.23.115.62@KRBTEST.COM
I20260620 18:38:07.073343 25230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:07.050710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.3:39111 (local address 127.23.115.62:40053)
0620 18:38:07.050894 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:07.050899 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:07.052215 (+ 1316us) server_negotiation.cc:407] Connection header received
0620 18:38:07.054459 (+ 2244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:07.054466 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:07.054560 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:07.054641 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:07.056906 (+ 2265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:07.057993 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:07.060795 (+ 2802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:07.061354 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:07.061448 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:07.066493 (+ 5045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:07.066501 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:07.066506 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:07.066549 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:07.069480 (+ 2931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:07.070494 (+ 1014us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:07.070500 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:07.070504 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:07.070661 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:07.071206 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:07.071209 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:07.071212 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:07.071565 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:07.071709 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:07.072698 (+ 989us) server_negotiation.cc:299] Negotiation successful
0620 18:38:07.072955 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 18:38:07.074844 25509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:07.051232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:40053 (local address 127.23.115.3:39111)
0620 18:38:07.052036 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:07.052088 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:07.054120 (+ 2032us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:07.055153 (+ 1033us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:07.055164 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:07.055901 (+ 737us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:07.056734 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:07.056762 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:07.058177 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:07.058183 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:07.060621 (+ 2438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:07.060638 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:07.061597 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:07.061605 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:07.061907 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:07.062742 (+ 835us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:07.062776 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:07.066209 (+ 3433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:07.069614 (+ 3405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:07.069627 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:07.069646 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:07.070342 (+ 696us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:07.070790 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:07.070795 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:07.070799 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:07.071085 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:07.071683 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:07.071691 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:07.072183 (+ 492us) client_negotiation.cc:769] Sending connection context
0620 18:38:07.072549 (+ 366us) client_negotiation.cc:240] Negotiation successful
0620 18:38:07.073162 (+ 613us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":318,"threads_started":1}
I20260620 18:38:07.087265 25485 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.3:45969
I20260620 18:38:07.087329 25619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.3:45969 every 8 connection(s)
I20260620 18:38:07.088460 25485 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980676253068-24012-0/minicluster-data/ts-2/data/info.pb
I20260620 18:38:07.090564 25620 heartbeater.cc:347] Connected to a master server at 127.23.115.62:40053
I20260620 18:38:07.090767 25620 heartbeater.cc:464] Registering TS with master...
I20260620 18:38:07.091396 25620 heartbeater.cc:511] Master 127.23.115.62:40053 requested a full tablet report, sending...
I20260620 18:38:07.092715 25134 ts_manager.cc:194] Registered new tserver with Master: 39e676f08fe246fa9d7590ab40edd09e (127.23.115.3:45969)
I20260620 18:38:07.093708 25134 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.115.3@KRBTEST.COM'} at 127.23.115.3:39111
I20260620 18:38:07.095706 25230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:07.081519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.3:54219 (local address 127.23.115.62:40053)
0620 18:38:07.082075 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:07.082081 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:07.082538 (+ 457us) server_negotiation.cc:407] Connection header received
0620 18:38:07.082728 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:07.082733 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:07.082829 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:07.082956 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:07.084728 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:07.085756 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:07.087771 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:07.088193 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:07.088298 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:07.090352 (+ 2054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:07.090358 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:07.090362 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:07.090397 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:07.093324 (+ 2927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:07.093925 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:07.093929 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:07.093932 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:07.094075 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:07.094457 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:07.094463 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:07.094467 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:07.094701 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:07.094803 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:07.095289 (+ 486us) server_negotiation.cc:299] Negotiation successful
0620 18:38:07.095439 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423}
I20260620 18:38:07.095940 25605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:07.081667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:40053 (local address 127.23.115.3:54219)
0620 18:38:07.082425 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:07.082447 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:07.082596 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:07.083005 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:07.083011 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:07.084060 (+ 1049us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:07.084549 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:07.084566 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:07.085900 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:07.085905 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:07.087640 (+ 1735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:07.087652 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:07.088362 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:07.088368 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:07.088526 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:07.089159 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:07.089181 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:07.090167 (+ 986us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:07.093460 (+ 3293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:07.093464 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:07.093467 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:07.093822 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:07.094205 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:07.094211 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:07.094215 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:07.094351 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:07.094810 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:07.094815 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:07.095018 (+ 203us) client_negotiation.cc:769] Sending connection context
0620 18:38:07.095189 (+ 171us) client_negotiation.cc:240] Negotiation successful
0620 18:38:07.095492 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":400,"threads_started":1}
I20260620 18:38:07.098214 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 25485
I20260620 18:38:07.098404 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980676253068-24012-0/minicluster-data/ts-2/wal/instance
I20260620 18:38:07.098877 24012 external_mini_cluster.cc:1428] Setting key 1b532473b46c9b1d6955d53cd14294be
I20260620 18:38:07.101531 24012 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:38:07.120651 25342 heartbeater.cc:503] Master 127.23.115.62:40053 was elected leader, sending a full tablet report...
Jun 20 18:38:07 dist-test-slave-d645 krb5kdc[25090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980687, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 18:38:07 dist-test-slave-d645 krb5kdc[25090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980687, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.115.62@KRBTEST.COM
I20260620 18:38:07.143270 25230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:07.129929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56558 (local address 127.23.115.62:40053)
0620 18:38:07.130083 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:07.130088 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:07.130479 (+ 391us) server_negotiation.cc:407] Connection header received
0620 18:38:07.130594 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:07.130598 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:07.130669 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:07.130769 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:07.132211 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:07.133045 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:07.134312 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:07.134763 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:07.134866 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:07.138480 (+ 3614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:07.138485 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:07.138489 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:07.138521 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:07.140895 (+ 2374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:07.141469 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:07.141473 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:07.141476 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:07.141588 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:07.141925 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:07.141930 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:07.141933 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:07.142194 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:07.142421 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:07.142688 (+ 267us) server_negotiation.cc:299] Negotiation successful
0620 18:38:07.142899 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 18:38:07.147135 25134 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56558:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:38:07.153643 25134 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:38:07.178723 25633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:07.168206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.1:41579 (local address 127.0.0.1:37678)
0620 18:38:07.168871 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:07.168907 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:07.169084 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:07.170486 (+ 1402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:07.170495 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:07.170529 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:38:07.170988 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:07.171001 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:07.173164 (+ 2163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:07.173171 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:07.175653 (+ 2482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:07.175671 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:07.177700 (+ 2029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:07.177707 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:07.177928 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:07.177988 (+ 60us) client_negotiation.cc:769] Sending connection context
0620 18:38:07.178118 (+ 130us) client_negotiation.cc:240] Negotiation successful
0620 18:38:07.178318 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":1236,"threads_started":1}
I20260620 18:38:07.180068 25634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:07.168773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37678 (local address 127.23.115.1:41579)
0620 18:38:07.169817 (+ 1044us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:07.169827 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:07.169856 (+ 29us) server_negotiation.cc:407] Connection header received
0620 18:38:07.169964 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:07.169973 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:07.170235 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:07.171199 (+ 964us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:38:07.171387 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:07.172987 (+ 1600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:07.175879 (+ 2892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:07.177452 (+ 1573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:07.179252 (+ 1800us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:07.179500 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:07.179576 (+ 76us) server_negotiation.cc:299] Negotiation successful
0620 18:38:07.179757 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":886,"thread_start_us":2180,"threads_started":1}
I20260620 18:38:07.184214 25272 tablet_service.cc:1467] Processing CreateTablet for tablet 991834a5920c4c3aaa5bcf2b6be2b9f2 (DEFAULT_TABLE table=test-table [id=8cf9b67cae754ecd919c60525cf8b637]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:38:07.185783 25272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 991834a5920c4c3aaa5bcf2b6be2b9f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:07.187455 25638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:07.170739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47942 (local address 127.23.115.3:45969)
0620 18:38:07.180522 (+ 9783us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:07.180530 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:07.180562 (+ 32us) server_negotiation.cc:407] Connection header received
0620 18:38:07.180669 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:07.180676 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:07.180919 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:07.181095 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:38:07.181783 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:07.183137 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:07.184835 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:07.186377 (+ 1542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:07.186509 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:07.186737 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:07.186967 (+ 230us) server_negotiation.cc:299] Negotiation successful
0620 18:38:07.187128 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9611,"thread_start_us":8337,"threads_started":1}
I20260620 18:38:07.187448 25637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:07.170122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.3:45969 (local address 127.0.0.1:47942)
0620 18:38:07.179986 (+ 9864us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:07.180001 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:07.180094 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:07.181122 (+ 1028us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:07.181128 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:07.181159 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:38:07.181529 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:07.181539 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:07.183264 (+ 1725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:07.183269 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:07.184636 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:07.184648 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:07.186573 (+ 1925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:07.186579 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:07.186778 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:07.186809 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 18:38:07.186888 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 18:38:07.187122 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9764,"thread_start_us":9610,"threads_started":1}
I20260620 18:38:07.189285 25633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:07.169444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.2:35569 (local address 127.0.0.1:49194)
0620 18:38:07.179498 (+ 10054us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:07.179519 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:07.179629 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:07.183013 (+ 3384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:07.183017 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:07.183042 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:38:07.183244 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:07.183255 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:07.185067 (+ 1812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:07.185072 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:07.186885 (+ 1813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:07.186899 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:07.188537 (+ 1638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:07.188543 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:07.188715 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:07.188748 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 18:38:07.188844 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 18:38:07.188977 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9943,"mutex_wait_us":3,"thread_start_us":11856,"threads_started":1}
I20260620 18:38:07.189311 25636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:07.169476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49194 (local address 127.23.115.2:35569)
0620 18:38:07.182464 (+ 12988us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:07.182472 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:07.182499 (+ 27us) server_negotiation.cc:407] Connection header received
0620 18:38:07.182582 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:07.182589 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:07.182797 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:07.183407 (+ 610us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:38:07.183581 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:07.184910 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:07.187126 (+ 2216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:07.188366 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:07.188495 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:07.188678 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:07.188896 (+ 218us) server_negotiation.cc:299] Negotiation successful
0620 18:38:07.189055 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12831,"thread_start_us":12846,"threads_started":1}
I20260620 18:38:07.190971 25550 tablet_service.cc:1467] Processing CreateTablet for tablet 991834a5920c4c3aaa5bcf2b6be2b9f2 (DEFAULT_TABLE table=test-table [id=8cf9b67cae754ecd919c60525cf8b637]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:38:07.192386 25550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 991834a5920c4c3aaa5bcf2b6be2b9f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:07.192762 25411 tablet_service.cc:1467] Processing CreateTablet for tablet 991834a5920c4c3aaa5bcf2b6be2b9f2 (DEFAULT_TABLE table=test-table [id=8cf9b67cae754ecd919c60525cf8b637]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:38:07.194166 25411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 991834a5920c4c3aaa5bcf2b6be2b9f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:07.197880 25639 tablet_bootstrap.cc:492] T 991834a5920c4c3aaa5bcf2b6be2b9f2 P 26f1de5d3e9744eebd1deb54b454f585: Bootstrap starting.
I20260620 18:38:07.200819 25639 tablet_bootstrap.cc:654] T 991834a5920c4c3aaa5bcf2b6be2b9f2 P 26f1de5d3e9744eebd1deb54b454f585: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:07.201757 25639 log.cc:824] T 991834a5920c4c3aaa5bcf2b6be2b9f2 P 26f1de5d3e9744eebd1deb54b454f585: Log is configured to *not* fsync() on all Append() calls
I20260620 18:38:07.205034 25639 tablet_bootstrap.cc:492] T 991834a5920c4c3aaa5bcf2b6be2b9f2 P 26f1de5d3e9744eebd1deb54b454f585: No bootstrap required, opened a new log
I20260620 18:38:07.205269 25639 ts_tablet_manager.cc:1397] T 991834a5920c4c3aaa5bcf2b6be2b9f2 P 26f1de5d3e9744eebd1deb54b454f585: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260620 18:38:07.206830 25641 tablet_bootstrap.cc:492] T 991834a5920c4c3aaa5bcf2b6be2b9f2 P 39e676f08fe246fa9d7590ab40edd09e: Bootstrap starting.
I20260620 18:38:07.207942 25642 tablet_bootstrap.cc:492] T 991834a5920c4c3aaa5bcf2b6be2b9f2 P afde26b99c8d4e2e8ac019f9fa3fb40c: Bootstrap starting.
I20260620 18:38:07.210487 25641 tablet_bootstrap.cc:654] T 991834a5920c4c3aaa5bcf2b6be2b9f2 P 39e676f08fe246fa9d7590ab40edd09e: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:07.210718 25642 tablet_bootstrap.cc:654] T 991834a5920c4c3aaa5bcf2b6be2b9f2 P afde26b99c8d4e2e8ac019f9fa3fb40c: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:07.211619 25642 log.cc:824] T 991834a5920c4c3aaa5bcf2b6be2b9f2 P afde26b99c8d4e2e8ac019f9fa3fb40c: Log is configured to *not* fsync() on all Append() calls
I20260620 18:38:07.211668 25641 log.cc:824] T 991834a5920c4c3aaa5bcf2b6be2b9f2 P 39e676f08fe246fa9d7590ab40edd09e: Log is configured to *not* fsync() on all Append() calls
I20260620 18:38:07.213928 25642 tablet_bootstrap.cc:492] T 991834a5920c4c3aaa5bcf2b6be2b9f2 P afde26b99c8d4e2e8ac019f9fa3fb40c: No bootstrap required, opened a new log
I20260620 18:38:07.214149 25642 ts_tablet_manager.cc:1397] T 991834a5920c4c3aaa5bcf2b6be2b9f2 P afde26b99c8d4e2e8ac019f9fa3fb40c: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260620 18:38:07.215427 25639 raft_consensus.cc:348] T 991834a5920c4c3aaa5bcf2b6be2b9f2 P 26f1de5d3e9744eebd1deb54b454f585 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26f1de5d3e9744eebd1deb54b454f585" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 41579 } } peers { permanent_uuid: "39e676f08fe246fa9d7590ab40edd09e" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 45969 } } peers { permanent_uuid: "afde26b99c8d4e2e8ac019f9fa3fb40c" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 35569 } }
I20260620 18:38:07.215936 25639 raft_consensus.cc:374] T 991834a5920c4c3aaa5bcf2b6be2b9f2 P 26f1de5d3e9744eebd1deb54b454f585 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:07.216099 25639 raft_consensus.cc:729] T 991834a5920c4c3aaa5bcf2b6be2b9f2 P 26f1de5d3e9744eebd1deb54b454f585 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26f1de5d3e9744eebd1deb54b454f585, State: Initialized, Role: FOLLOWER
I20260620 18:38:07.216759 25639 consensus_queue.cc:260] T 991834a5920c4c3aaa5bcf2b6be2b9f2 P 26f1de5d3e9744eebd1deb54b454f585 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26f1de5d3e9744eebd1deb54b454f585" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 41579 } } peers { permanent_uuid: "39e676f08fe246fa9d7590ab40edd09e" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 45969 } } peers { permanent_uuid: "afde26b99c8d4e2e8ac019f9fa3fb40c" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 35569 } }
I20260620 18:38:07.218478 25639 ts_tablet_manager.cc:1428] T 991834a5920c4c3aaa5bcf2b6be2b9f2 P 26f1de5d3e9744eebd1deb54b454f585: Time spent starting tablet: real 0.013s user 0.009s sys 0.005s
I20260620 18:38:07.222107 25642 raft_consensus.cc:348] T 991834a5920c4c3aaa5bcf2b6be2b9f2 P afde26b99c8d4e2e8ac019f9fa3fb40c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26f1de5d3e9744eebd1deb54b454f585" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 41579 } } peers { permanent_uuid: "39e676f08fe246fa9d7590ab40edd09e" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 45969 } } peers { permanent_uuid: "afde26b99c8d4e2e8ac019f9fa3fb40c" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 35569 } }
I20260620 18:38:07.222529 25642 raft_consensus.cc:374] T 991834a5920c4c3aaa5bcf2b6be2b9f2 P afde26b99c8d4e2e8ac019f9fa3fb40c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:07.222627 25642 raft_consensus.cc:729] T 991834a5920c4c3aaa5bcf2b6be2b9f2 P afde26b99c8d4e2e8ac019f9fa3fb40c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afde26b99c8d4e2e8ac019f9fa3fb40c, State: Initialized, Role: FOLLOWER
I20260620 18:38:07.223244 25642 consensus_queue.cc:260] T 991834a5920c4c3aaa5bcf2b6be2b9f2 P afde26b99c8d4e2e8ac019f9fa3fb40c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26f1de5d3e9744eebd1deb54b454f585" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 41579 } } peers { permanent_uuid: "39e676f08fe246fa9d7590ab40edd09e" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 45969 } } peers { permanent_uuid: "afde26b99c8d4e2e8ac019f9fa3fb40c" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 35569 } }
I20260620 18:38:07.224220 25481 heartbeater.cc:503] Master 127.23.115.62:40053 was elected leader, sending a full tablet report...
I20260620 18:38:07.224815 25641 tablet_bootstrap.cc:492] T 991834a5920c4c3aaa5bcf2b6be2b9f2 P 39e676f08fe246fa9d7590ab40edd09e: No bootstrap required, opened a new log
I20260620 18:38:07.224947 25642 ts_tablet_manager.cc:1428] T 991834a5920c4c3aaa5bcf2b6be2b9f2 P afde26b99c8d4e2e8ac019f9fa3fb40c: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260620 18:38:07.225098 25641 ts_tablet_manager.cc:1397] T 991834a5920c4c3aaa5bcf2b6be2b9f2 P 39e676f08fe246fa9d7590ab40edd09e: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.010s
I20260620 18:38:07.232690 25641 raft_consensus.cc:348] T 991834a5920c4c3aaa5bcf2b6be2b9f2 P 39e676f08fe246fa9d7590ab40edd09e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26f1de5d3e9744eebd1deb54b454f585" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 41579 } } peers { permanent_uuid: "39e676f08fe246fa9d7590ab40edd09e" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 45969 } } peers { permanent_uuid: "afde26b99c8d4e2e8ac019f9fa3fb40c" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 35569 } }
I20260620 18:38:07.233044 25641 raft_consensus.cc:374] T 991834a5920c4c3aaa5bcf2b6be2b9f2 P 39e676f08fe246fa9d7590ab40edd09e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:07.233141 25641 raft_consensus.cc:729] T 991834a5920c4c3aaa5bcf2b6be2b9f2 P 39e676f08fe246fa9d7590ab40edd09e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39e676f08fe246fa9d7590ab40edd09e, State: Initialized, Role: FOLLOWER
I20260620 18:38:07.233644 25641 consensus_queue.cc:260] T 991834a5920c4c3aaa5bcf2b6be2b9f2 P 39e676f08fe246fa9d7590ab40edd09e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26f1de5d3e9744eebd1deb54b454f585" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 41579 } } peers { permanent_uuid: "39e676f08fe246fa9d7590ab40edd09e" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 45969 } } peers { permanent_uuid: "afde26b99c8d4e2e8ac019f9fa3fb40c" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 35569 } }
I20260620 18:38:07.234572 25620 heartbeater.cc:503] Master 127.23.115.62:40053 was elected leader, sending a full tablet report...
I20260620 18:38:07.234997 25641 ts_tablet_manager.cc:1428] T 991834a5920c4c3aaa5bcf2b6be2b9f2 P 39e676f08fe246fa9d7590ab40edd09e: Time spent starting tablet: real 0.010s user 0.012s sys 0.000s
I20260620 18:38:07.282080 25647 raft_consensus.cc:482] T 991834a5920c4c3aaa5bcf2b6be2b9f2 P 39e676f08fe246fa9d7590ab40edd09e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:38:07.282393 25647 raft_consensus.cc:504] T 991834a5920c4c3aaa5bcf2b6be2b9f2 P 39e676f08fe246fa9d7590ab40edd09e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26f1de5d3e9744eebd1deb54b454f585" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 41579 } } peers { permanent_uuid: "39e676f08fe246fa9d7590ab40edd09e" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 45969 } } peers { permanent_uuid: "afde26b99c8d4e2e8ac019f9fa3fb40c" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 35569 } }
I20260620 18:38:07.284102 25647 leader_election.cc:288] T 991834a5920c4c3aaa5bcf2b6be2b9f2 P 39e676f08fe246fa9d7590ab40edd09e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26f1de5d3e9744eebd1deb54b454f585 (127.23.115.1:41579), afde26b99c8d4e2e8ac019f9fa3fb40c (127.23.115.2:35569)
I20260620 18:38:07.290014 25636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:07.284494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.3:56337 (local address 127.23.115.2:35569)
0620 18:38:07.284813 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:07.284819 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:07.284838 (+ 19us) server_negotiation.cc:407] Connection header received
0620 18:38:07.285032 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:07.285036 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:07.285096 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:07.285171 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:38:07.285597 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:07.286419 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:07.288396 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:07.289144 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:07.289305 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:07.289382 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:07.289615 (+ 233us) server_negotiation.cc:299] Negotiation successful
0620 18:38:07.289745 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192}
I20260620 18:38:07.290012 25509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:07.284493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.2:35569 (local address 127.23.115.3:56337)
0620 18:38:07.284757 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:07.284779 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:07.284905 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:07.285213 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:07.285217 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:07.285248 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:38:07.285427 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:07.285435 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:07.286569 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:07.286574 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:07.288211 (+ 1637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:07.288231 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:07.289291 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:07.289297 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:07.289466 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:07.289491 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 18:38:07.289572 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 18:38:07.289710 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129}
I20260620 18:38:07.290313 25634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:07.284536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.3:50043 (local address 127.23.115.1:41579)
0620 18:38:07.284728 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:07.284735 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:07.285346 (+ 611us) server_negotiation.cc:407] Connection header received
0620 18:38:07.285518 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:07.285522 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:07.285587 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:07.285661 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:38:07.286147 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:07.286833 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:07.288703 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:07.289482 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:07.289625 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:07.289695 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:07.289959 (+ 264us) server_negotiation.cc:299] Negotiation successful
0620 18:38:07.290090 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 18:38:07.290374 25648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:07.284530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.1:41579 (local address 127.23.115.3:50043)
0620 18:38:07.285247 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:07.285262 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:07.285392 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:07.285684 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:07.285688 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:07.285710 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:38:07.285999 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:07.286007 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:07.286979 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:07.286983 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:07.288545 (+ 1562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:07.288557 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:07.289684 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:07.289690 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:07.289813 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:07.289836 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 18:38:07.289931 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 18:38:07.290054 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":404,"threads_started":1}
I20260620 18:38:07.290908 25436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "991834a5920c4c3aaa5bcf2b6be2b9f2" candidate_uuid: "39e676f08fe246fa9d7590ab40edd09e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "afde26b99c8d4e2e8ac019f9fa3fb40c" is_pre_election: true
I20260620 18:38:07.291610 25436 raft_consensus.cc:2454] T 991834a5920c4c3aaa5bcf2b6be2b9f2 P afde26b99c8d4e2e8ac019f9fa3fb40c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39e676f08fe246fa9d7590ab40edd09e in term 0.
I20260620 18:38:07.291700 25292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "991834a5920c4c3aaa5bcf2b6be2b9f2" candidate_uuid: "39e676f08fe246fa9d7590ab40edd09e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26f1de5d3e9744eebd1deb54b454f585" is_pre_election: true
I20260620 18:38:07.292147 25292 raft_consensus.cc:2454] T 991834a5920c4c3aaa5bcf2b6be2b9f2 P 26f1de5d3e9744eebd1deb54b454f585 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39e676f08fe246fa9d7590ab40edd09e in term 0.
I20260620 18:38:07.292222 25502 leader_election.cc:302] T 991834a5920c4c3aaa5bcf2b6be2b9f2 P 39e676f08fe246fa9d7590ab40edd09e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39e676f08fe246fa9d7590ab40edd09e, afde26b99c8d4e2e8ac019f9fa3fb40c; no voters:
I20260620 18:38:07.292615 25647 raft_consensus.cc:2781] T 991834a5920c4c3aaa5bcf2b6be2b9f2 P 39e676f08fe246fa9d7590ab40edd09e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:38:07.292768 25647 raft_consensus.cc:482] T 991834a5920c4c3aaa5bcf2b6be2b9f2 P 39e676f08fe246fa9d7590ab40edd09e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:38:07.292873 25647 raft_consensus.cc:3037] T 991834a5920c4c3aaa5bcf2b6be2b9f2 P 39e676f08fe246fa9d7590ab40edd09e [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:07.295066 25647 raft_consensus.cc:504] T 991834a5920c4c3aaa5bcf2b6be2b9f2 P 39e676f08fe246fa9d7590ab40edd09e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26f1de5d3e9744eebd1deb54b454f585" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 41579 } } peers { permanent_uuid: "39e676f08fe246fa9d7590ab40edd09e" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 45969 } } peers { permanent_uuid: "afde26b99c8d4e2e8ac019f9fa3fb40c" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 35569 } }
I20260620 18:38:07.295671 25647 leader_election.cc:288] T 991834a5920c4c3aaa5bcf2b6be2b9f2 P 39e676f08fe246fa9d7590ab40edd09e [CANDIDATE]: Term 1 election: Requested vote from peers 26f1de5d3e9744eebd1deb54b454f585 (127.23.115.1:41579), afde26b99c8d4e2e8ac019f9fa3fb40c (127.23.115.2:35569)
I20260620 18:38:07.296100 25292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "991834a5920c4c3aaa5bcf2b6be2b9f2" candidate_uuid: "39e676f08fe246fa9d7590ab40edd09e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26f1de5d3e9744eebd1deb54b454f585"
I20260620 18:38:07.296228 25436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "991834a5920c4c3aaa5bcf2b6be2b9f2" candidate_uuid: "39e676f08fe246fa9d7590ab40edd09e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "afde26b99c8d4e2e8ac019f9fa3fb40c"
I20260620 18:38:07.296299 25292 raft_consensus.cc:3037] T 991834a5920c4c3aaa5bcf2b6be2b9f2 P 26f1de5d3e9744eebd1deb54b454f585 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:07.296440 25436 raft_consensus.cc:3037] T 991834a5920c4c3aaa5bcf2b6be2b9f2 P afde26b99c8d4e2e8ac019f9fa3fb40c [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:07.298529 25292 raft_consensus.cc:2454] T 991834a5920c4c3aaa5bcf2b6be2b9f2 P 26f1de5d3e9744eebd1deb54b454f585 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39e676f08fe246fa9d7590ab40edd09e in term 1.
I20260620 18:38:07.298631 25436 raft_consensus.cc:2454] T 991834a5920c4c3aaa5bcf2b6be2b9f2 P afde26b99c8d4e2e8ac019f9fa3fb40c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39e676f08fe246fa9d7590ab40edd09e in term 1.
I20260620 18:38:07.298960 25501 leader_election.cc:302] T 991834a5920c4c3aaa5bcf2b6be2b9f2 P 39e676f08fe246fa9d7590ab40edd09e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26f1de5d3e9744eebd1deb54b454f585, 39e676f08fe246fa9d7590ab40edd09e; no voters:
I20260620 18:38:07.299239 25647 raft_consensus.cc:2781] T 991834a5920c4c3aaa5bcf2b6be2b9f2 P 39e676f08fe246fa9d7590ab40edd09e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:38:07.299623 25647 raft_consensus.cc:686] T 991834a5920c4c3aaa5bcf2b6be2b9f2 P 39e676f08fe246fa9d7590ab40edd09e [term 1 LEADER]: Becoming Leader. State: Replica: 39e676f08fe246fa9d7590ab40edd09e, State: Running, Role: LEADER
I20260620 18:38:07.300056 25647 consensus_queue.cc:237] T 991834a5920c4c3aaa5bcf2b6be2b9f2 P 39e676f08fe246fa9d7590ab40edd09e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26f1de5d3e9744eebd1deb54b454f585" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 41579 } } peers { permanent_uuid: "39e676f08fe246fa9d7590ab40edd09e" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 45969 } } peers { permanent_uuid: "afde26b99c8d4e2e8ac019f9fa3fb40c" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 35569 } }
I20260620 18:38:07.304731 25134 catalog_manager.cc:5495] T 991834a5920c4c3aaa5bcf2b6be2b9f2 P 39e676f08fe246fa9d7590ab40edd09e reported cstate change: term changed from 0 to 1, leader changed from <none> to 39e676f08fe246fa9d7590ab40edd09e (127.23.115.3). New cstate: current_term: 1 leader_uuid: "39e676f08fe246fa9d7590ab40edd09e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26f1de5d3e9744eebd1deb54b454f585" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 41579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39e676f08fe246fa9d7590ab40edd09e" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 45969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "afde26b99c8d4e2e8ac019f9fa3fb40c" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 35569 } health_report { overall_health: UNKNOWN } } }
W20260620 18:38:07.336664 25482 tablet.cc:2369] T 991834a5920c4c3aaa5bcf2b6be2b9f2 P afde26b99c8d4e2e8ac019f9fa3fb40c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:38:07.339947 25621 tablet.cc:2369] T 991834a5920c4c3aaa5bcf2b6be2b9f2 P 39e676f08fe246fa9d7590ab40edd09e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:38:07.354056 25638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:07.348671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47950 (local address 127.23.115.3:45969)
0620 18:38:07.348857 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:07.348862 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:07.348961 (+ 99us) server_negotiation.cc:407] Connection header received
0620 18:38:07.349281 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:07.349285 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:07.349367 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:07.349591 (+ 224us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:38:07.349929 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:07.350620 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:07.352020 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:07.352449 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:07.352545 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:07.352994 (+ 449us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:38:07.353491 (+ 497us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:38:07.353603 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:07.353788 (+ 185us) server_negotiation.cc:299] Negotiation successful
0620 18:38:07.353856 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 18:38:07.362269 25292 raft_consensus.cc:1261] T 991834a5920c4c3aaa5bcf2b6be2b9f2 P 26f1de5d3e9744eebd1deb54b454f585 [term 1 FOLLOWER]: Refusing update from remote peer 39e676f08fe246fa9d7590ab40edd09e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:38:07.362591 25436 raft_consensus.cc:1261] T 991834a5920c4c3aaa5bcf2b6be2b9f2 P afde26b99c8d4e2e8ac019f9fa3fb40c [term 1 FOLLOWER]: Refusing update from remote peer 39e676f08fe246fa9d7590ab40edd09e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:38:07.363209 25649 consensus_queue.cc:1035] T 991834a5920c4c3aaa5bcf2b6be2b9f2 P 39e676f08fe246fa9d7590ab40edd09e [LEADER]: Connected to new peer: Peer: permanent_uuid: "26f1de5d3e9744eebd1deb54b454f585" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 41579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:38:07.363596 25647 consensus_queue.cc:1035] T 991834a5920c4c3aaa5bcf2b6be2b9f2 P 39e676f08fe246fa9d7590ab40edd09e [LEADER]: Connected to new peer: Peer: permanent_uuid: "afde26b99c8d4e2e8ac019f9fa3fb40c" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 35569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 18:38:07.363785 25343 tablet.cc:2369] T 991834a5920c4c3aaa5bcf2b6be2b9f2 P 26f1de5d3e9744eebd1deb54b454f585: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:38:07.391991 25135 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56558:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:38:07.392220 25135 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56558:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:38:07.395722 25135 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d9e48bfe423541ea965d5203d2ce8dbe: Sending DeleteTablet for 3 replicas of tablet 991834a5920c4c3aaa5bcf2b6be2b9f2
I20260620 18:38:07.396935 25272 tablet_service.cc:1514] Processing DeleteTablet for tablet 991834a5920c4c3aaa5bcf2b6be2b9f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:38:07 UTC) from {username='kudu', principal='kudu/127.23.115.62@KRBTEST.COM'} at 127.0.0.1:37678
I20260620 18:38:07.397171 25411 tablet_service.cc:1514] Processing DeleteTablet for tablet 991834a5920c4c3aaa5bcf2b6be2b9f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:38:07 UTC) from {username='kudu', principal='kudu/127.23.115.62@KRBTEST.COM'} at 127.0.0.1:49194
I20260620 18:38:07.397202 25550 tablet_service.cc:1514] Processing DeleteTablet for tablet 991834a5920c4c3aaa5bcf2b6be2b9f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:38:07 UTC) from {username='kudu', principal='kudu/127.23.115.62@KRBTEST.COM'} at 127.0.0.1:47942
I20260620 18:38:07.399245 25135 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56558:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:38:07.399940 25135 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:38:07.407748 25271 tablet_service.cc:1467] Processing CreateTablet for tablet 0d61864f358a4a7a933222df4294b227 (DEFAULT_TABLE table=test-table [id=ad7b132f700f4d1eb9b309454345e5c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:38:07.407907 25549 tablet_service.cc:1467] Processing CreateTablet for tablet 0d61864f358a4a7a933222df4294b227 (DEFAULT_TABLE table=test-table [id=ad7b132f700f4d1eb9b309454345e5c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:38:07.408366 25271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d61864f358a4a7a933222df4294b227. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:07.408463 25549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d61864f358a4a7a933222df4294b227. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:07.408754 25663 tablet_replica.cc:330] T 991834a5920c4c3aaa5bcf2b6be2b9f2 P afde26b99c8d4e2e8ac019f9fa3fb40c: stopping tablet replica
I20260620 18:38:07.409498 25663 raft_consensus.cc:2229] T 991834a5920c4c3aaa5bcf2b6be2b9f2 P afde26b99c8d4e2e8ac019f9fa3fb40c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:38:07.410629 25663 raft_consensus.cc:2258] T 991834a5920c4c3aaa5bcf2b6be2b9f2 P afde26b99c8d4e2e8ac019f9fa3fb40c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:38:07.411599 25662 tablet_replica.cc:330] T 991834a5920c4c3aaa5bcf2b6be2b9f2 P 26f1de5d3e9744eebd1deb54b454f585: stopping tablet replica
I20260620 18:38:07.412655 25662 raft_consensus.cc:2229] T 991834a5920c4c3aaa5bcf2b6be2b9f2 P 26f1de5d3e9744eebd1deb54b454f585 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:38:07.413123 25410 tablet_service.cc:1467] Processing CreateTablet for tablet 0d61864f358a4a7a933222df4294b227 (DEFAULT_TABLE table=test-table [id=ad7b132f700f4d1eb9b309454345e5c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:38:07.413612 25410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d61864f358a4a7a933222df4294b227. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:07.414094 25662 raft_consensus.cc:2258] T 991834a5920c4c3aaa5bcf2b6be2b9f2 P 26f1de5d3e9744eebd1deb54b454f585 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:38:07.415117 25641 tablet_bootstrap.cc:492] T 0d61864f358a4a7a933222df4294b227 P 39e676f08fe246fa9d7590ab40edd09e: Bootstrap starting.
I20260620 18:38:07.414861 25663 ts_tablet_manager.cc:1905] T 991834a5920c4c3aaa5bcf2b6be2b9f2 P afde26b99c8d4e2e8ac019f9fa3fb40c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:38:07.416742 25641 tablet_bootstrap.cc:654] T 0d61864f358a4a7a933222df4294b227 P 39e676f08fe246fa9d7590ab40edd09e: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:07.417976 25662 ts_tablet_manager.cc:1905] T 991834a5920c4c3aaa5bcf2b6be2b9f2 P 26f1de5d3e9744eebd1deb54b454f585: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:38:07.418571 25664 tablet_replica.cc:330] T 991834a5920c4c3aaa5bcf2b6be2b9f2 P 39e676f08fe246fa9d7590ab40edd09e: stopping tablet replica
I20260620 18:38:07.419070 25664 raft_consensus.cc:2229] T 991834a5920c4c3aaa5bcf2b6be2b9f2 P 39e676f08fe246fa9d7590ab40edd09e [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:38:07.423597 25664 raft_consensus.cc:2258] T 991834a5920c4c3aaa5bcf2b6be2b9f2 P 39e676f08fe246fa9d7590ab40edd09e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:38:07.424342 25662 ts_tablet_manager.cc:1918] T 991834a5920c4c3aaa5bcf2b6be2b9f2 P 26f1de5d3e9744eebd1deb54b454f585: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:38:07.424486 25662 log.cc:1196] T 991834a5920c4c3aaa5bcf2b6be2b9f2 P 26f1de5d3e9744eebd1deb54b454f585: Deleting WAL directory at /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980676253068-24012-0/minicluster-data/ts-0/wal/wals/991834a5920c4c3aaa5bcf2b6be2b9f2
I20260620 18:38:07.424993 25662 ts_tablet_manager.cc:1939] T 991834a5920c4c3aaa5bcf2b6be2b9f2 P 26f1de5d3e9744eebd1deb54b454f585: Deleting consensus metadata
I20260620 18:38:07.425218 25642 tablet_bootstrap.cc:492] T 0d61864f358a4a7a933222df4294b227 P afde26b99c8d4e2e8ac019f9fa3fb40c: Bootstrap starting.
I20260620 18:38:07.425920 25664 ts_tablet_manager.cc:1905] T 991834a5920c4c3aaa5bcf2b6be2b9f2 P 39e676f08fe246fa9d7590ab40edd09e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:38:07.426204 25663 ts_tablet_manager.cc:1918] T 991834a5920c4c3aaa5bcf2b6be2b9f2 P afde26b99c8d4e2e8ac019f9fa3fb40c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:38:07.426399 25663 log.cc:1196] T 991834a5920c4c3aaa5bcf2b6be2b9f2 P afde26b99c8d4e2e8ac019f9fa3fb40c: Deleting WAL directory at /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980676253068-24012-0/minicluster-data/ts-1/wal/wals/991834a5920c4c3aaa5bcf2b6be2b9f2
I20260620 18:38:07.426692 25119 catalog_manager.cc:4841] TS 26f1de5d3e9744eebd1deb54b454f585 (127.23.115.1:41579): tablet 991834a5920c4c3aaa5bcf2b6be2b9f2 (table test-table [id=8cf9b67cae754ecd919c60525cf8b637]) successfully deleted
I20260620 18:38:07.426951 25642 tablet_bootstrap.cc:654] T 0d61864f358a4a7a933222df4294b227 P afde26b99c8d4e2e8ac019f9fa3fb40c: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:07.427071 25663 ts_tablet_manager.cc:1939] T 991834a5920c4c3aaa5bcf2b6be2b9f2 P afde26b99c8d4e2e8ac019f9fa3fb40c: Deleting consensus metadata
I20260620 18:38:07.428010 25639 tablet_bootstrap.cc:492] T 0d61864f358a4a7a933222df4294b227 P 26f1de5d3e9744eebd1deb54b454f585: Bootstrap starting.
I20260620 18:38:07.429239 25120 catalog_manager.cc:4841] TS afde26b99c8d4e2e8ac019f9fa3fb40c (127.23.115.2:35569): tablet 991834a5920c4c3aaa5bcf2b6be2b9f2 (table test-table [id=8cf9b67cae754ecd919c60525cf8b637]) successfully deleted
I20260620 18:38:07.429458 25639 tablet_bootstrap.cc:654] T 0d61864f358a4a7a933222df4294b227 P 26f1de5d3e9744eebd1deb54b454f585: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:07.434186 25642 tablet_bootstrap.cc:492] T 0d61864f358a4a7a933222df4294b227 P afde26b99c8d4e2e8ac019f9fa3fb40c: No bootstrap required, opened a new log
I20260620 18:38:07.434340 25642 ts_tablet_manager.cc:1397] T 0d61864f358a4a7a933222df4294b227 P afde26b99c8d4e2e8ac019f9fa3fb40c: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.005s
I20260620 18:38:07.435122 25642 raft_consensus.cc:348] T 0d61864f358a4a7a933222df4294b227 P afde26b99c8d4e2e8ac019f9fa3fb40c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26f1de5d3e9744eebd1deb54b454f585" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 41579 } } peers { permanent_uuid: "afde26b99c8d4e2e8ac019f9fa3fb40c" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 35569 } } peers { permanent_uuid: "39e676f08fe246fa9d7590ab40edd09e" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 45969 } }
I20260620 18:38:07.435333 25642 raft_consensus.cc:374] T 0d61864f358a4a7a933222df4294b227 P afde26b99c8d4e2e8ac019f9fa3fb40c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:07.435413 25642 raft_consensus.cc:729] T 0d61864f358a4a7a933222df4294b227 P afde26b99c8d4e2e8ac019f9fa3fb40c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afde26b99c8d4e2e8ac019f9fa3fb40c, State: Initialized, Role: FOLLOWER
I20260620 18:38:07.435672 25642 consensus_queue.cc:260] T 0d61864f358a4a7a933222df4294b227 P afde26b99c8d4e2e8ac019f9fa3fb40c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26f1de5d3e9744eebd1deb54b454f585" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 41579 } } peers { permanent_uuid: "afde26b99c8d4e2e8ac019f9fa3fb40c" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 35569 } } peers { permanent_uuid: "39e676f08fe246fa9d7590ab40edd09e" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 45969 } }
I20260620 18:38:07.436249 25642 ts_tablet_manager.cc:1428] T 0d61864f358a4a7a933222df4294b227 P afde26b99c8d4e2e8ac019f9fa3fb40c: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260620 18:38:07.436542 25639 tablet_bootstrap.cc:492] T 0d61864f358a4a7a933222df4294b227 P 26f1de5d3e9744eebd1deb54b454f585: No bootstrap required, opened a new log
I20260620 18:38:07.436686 25639 ts_tablet_manager.cc:1397] T 0d61864f358a4a7a933222df4294b227 P 26f1de5d3e9744eebd1deb54b454f585: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260620 18:38:07.437438 25639 raft_consensus.cc:348] T 0d61864f358a4a7a933222df4294b227 P 26f1de5d3e9744eebd1deb54b454f585 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26f1de5d3e9744eebd1deb54b454f585" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 41579 } } peers { permanent_uuid: "afde26b99c8d4e2e8ac019f9fa3fb40c" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 35569 } } peers { permanent_uuid: "39e676f08fe246fa9d7590ab40edd09e" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 45969 } }
I20260620 18:38:07.437656 25639 raft_consensus.cc:374] T 0d61864f358a4a7a933222df4294b227 P 26f1de5d3e9744eebd1deb54b454f585 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:07.437740 25639 raft_consensus.cc:729] T 0d61864f358a4a7a933222df4294b227 P 26f1de5d3e9744eebd1deb54b454f585 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26f1de5d3e9744eebd1deb54b454f585, State: Initialized, Role: FOLLOWER
I20260620 18:38:07.437939 25664 ts_tablet_manager.cc:1918] T 991834a5920c4c3aaa5bcf2b6be2b9f2 P 39e676f08fe246fa9d7590ab40edd09e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:38:07.438059 25664 log.cc:1196] T 991834a5920c4c3aaa5bcf2b6be2b9f2 P 39e676f08fe246fa9d7590ab40edd09e: Deleting WAL directory at /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980676253068-24012-0/minicluster-data/ts-2/wal/wals/991834a5920c4c3aaa5bcf2b6be2b9f2
I20260620 18:38:07.437992 25639 consensus_queue.cc:260] T 0d61864f358a4a7a933222df4294b227 P 26f1de5d3e9744eebd1deb54b454f585 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26f1de5d3e9744eebd1deb54b454f585" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 41579 } } peers { permanent_uuid: "afde26b99c8d4e2e8ac019f9fa3fb40c" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 35569 } } peers { permanent_uuid: "39e676f08fe246fa9d7590ab40edd09e" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 45969 } }
I20260620 18:38:07.438634 25664 ts_tablet_manager.cc:1939] T 991834a5920c4c3aaa5bcf2b6be2b9f2 P 39e676f08fe246fa9d7590ab40edd09e: Deleting consensus metadata
I20260620 18:38:07.439074 25639 ts_tablet_manager.cc:1428] T 0d61864f358a4a7a933222df4294b227 P 26f1de5d3e9744eebd1deb54b454f585: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 18:38:07.441529 25119 catalog_manager.cc:4841] TS 39e676f08fe246fa9d7590ab40edd09e (127.23.115.3:45969): tablet 991834a5920c4c3aaa5bcf2b6be2b9f2 (table test-table [id=8cf9b67cae754ecd919c60525cf8b637]) successfully deleted
I20260620 18:38:07.442281 25641 tablet_bootstrap.cc:492] T 0d61864f358a4a7a933222df4294b227 P 39e676f08fe246fa9d7590ab40edd09e: No bootstrap required, opened a new log
I20260620 18:38:07.442456 25641 ts_tablet_manager.cc:1397] T 0d61864f358a4a7a933222df4294b227 P 39e676f08fe246fa9d7590ab40edd09e: Time spent bootstrapping tablet: real 0.027s user 0.005s sys 0.008s
I20260620 18:38:07.443161 25641 raft_consensus.cc:348] T 0d61864f358a4a7a933222df4294b227 P 39e676f08fe246fa9d7590ab40edd09e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26f1de5d3e9744eebd1deb54b454f585" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 41579 } } peers { permanent_uuid: "afde26b99c8d4e2e8ac019f9fa3fb40c" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 35569 } } peers { permanent_uuid: "39e676f08fe246fa9d7590ab40edd09e" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 45969 } }
I20260620 18:38:07.443337 25641 raft_consensus.cc:374] T 0d61864f358a4a7a933222df4294b227 P 39e676f08fe246fa9d7590ab40edd09e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:07.443432 25641 raft_consensus.cc:729] T 0d61864f358a4a7a933222df4294b227 P 39e676f08fe246fa9d7590ab40edd09e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39e676f08fe246fa9d7590ab40edd09e, State: Initialized, Role: FOLLOWER
I20260620 18:38:07.443709 25641 consensus_queue.cc:260] T 0d61864f358a4a7a933222df4294b227 P 39e676f08fe246fa9d7590ab40edd09e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26f1de5d3e9744eebd1deb54b454f585" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 41579 } } peers { permanent_uuid: "afde26b99c8d4e2e8ac019f9fa3fb40c" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 35569 } } peers { permanent_uuid: "39e676f08fe246fa9d7590ab40edd09e" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 45969 } }
I20260620 18:38:07.444186 25641 ts_tablet_manager.cc:1428] T 0d61864f358a4a7a933222df4294b227 P 39e676f08fe246fa9d7590ab40edd09e: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260620 18:38:07.462867 25646 raft_consensus.cc:482] T 0d61864f358a4a7a933222df4294b227 P afde26b99c8d4e2e8ac019f9fa3fb40c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:38:07.463160 25646 raft_consensus.cc:504] T 0d61864f358a4a7a933222df4294b227 P afde26b99c8d4e2e8ac019f9fa3fb40c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26f1de5d3e9744eebd1deb54b454f585" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 41579 } } peers { permanent_uuid: "afde26b99c8d4e2e8ac019f9fa3fb40c" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 35569 } } peers { permanent_uuid: "39e676f08fe246fa9d7590ab40edd09e" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 45969 } }
I20260620 18:38:07.464689 25646 leader_election.cc:288] T 0d61864f358a4a7a933222df4294b227 P afde26b99c8d4e2e8ac019f9fa3fb40c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26f1de5d3e9744eebd1deb54b454f585 (127.23.115.1:41579), 39e676f08fe246fa9d7590ab40edd09e (127.23.115.3:45969)
I20260620 18:38:07.471909 25634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:07.465084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.2:39273 (local address 127.23.115.1:41579)
0620 18:38:07.465254 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:07.465261 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:07.465609 (+ 348us) server_negotiation.cc:407] Connection header received
0620 18:38:07.465885 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:07.465891 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:07.465967 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:07.466032 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:38:07.466809 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:07.467771 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:07.470030 (+ 2259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:07.470856 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:07.471029 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:07.471107 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:07.471464 (+ 357us) server_negotiation.cc:299] Negotiation successful
0620 18:38:07.471631 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 18:38:07.471939 25668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:07.464984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.1:41579 (local address 127.23.115.2:39273)
0620 18:38:07.465514 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:07.465530 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:07.465724 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:07.466275 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:07.466280 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:07.466305 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:38:07.466674 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:07.466686 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:07.467892 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:07.467897 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:07.469877 (+ 1980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:07.469893 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:07.471095 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:07.471100 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:07.471282 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:07.471316 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 18:38:07.471411 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 18:38:07.471550 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"mutex_wait_us":4,"thread_start_us":292,"threads_started":1}
I20260620 18:38:07.472856 25292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d61864f358a4a7a933222df4294b227" candidate_uuid: "afde26b99c8d4e2e8ac019f9fa3fb40c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26f1de5d3e9744eebd1deb54b454f585" is_pre_election: true
I20260620 18:38:07.473093 25292 raft_consensus.cc:2454] T 0d61864f358a4a7a933222df4294b227 P 26f1de5d3e9744eebd1deb54b454f585 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate afde26b99c8d4e2e8ac019f9fa3fb40c in term 0.
I20260620 18:38:07.473531 25638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:07.465803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.2:50729 (local address 127.23.115.3:45969)
0620 18:38:07.465969 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:07.465974 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:07.466520 (+ 546us) server_negotiation.cc:407] Connection header received
0620 18:38:07.466697 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:07.466702 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:07.466785 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:07.466898 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:38:07.467470 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:07.468607 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:07.471250 (+ 2643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:07.472475 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:07.472655 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:07.472726 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:07.473109 (+ 383us) server_negotiation.cc:299] Negotiation successful
0620 18:38:07.473248 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 18:38:07.473587 25669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:07.465702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.3:45969 (local address 127.23.115.2:50729)
0620 18:38:07.466410 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:07.466427 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:07.466546 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:07.466920 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:07.466924 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:07.466971 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:38:07.467309 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:07.467321 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:07.468782 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:07.468787 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:07.471082 (+ 2295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:07.471104 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:07.472686 (+ 1582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:07.472693 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:07.472887 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:07.472923 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 18:38:07.473021 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 18:38:07.473213 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"mutex_wait_us":39,"thread_start_us":396,"threads_started":1}
I20260620 18:38:07.473651 25362 leader_election.cc:302] T 0d61864f358a4a7a933222df4294b227 P afde26b99c8d4e2e8ac019f9fa3fb40c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26f1de5d3e9744eebd1deb54b454f585, afde26b99c8d4e2e8ac019f9fa3fb40c; no voters:
I20260620 18:38:07.474102 25646 raft_consensus.cc:2781] T 0d61864f358a4a7a933222df4294b227 P afde26b99c8d4e2e8ac019f9fa3fb40c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:38:07.474234 25646 raft_consensus.cc:482] T 0d61864f358a4a7a933222df4294b227 P afde26b99c8d4e2e8ac019f9fa3fb40c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:38:07.474303 25646 raft_consensus.cc:3037] T 0d61864f358a4a7a933222df4294b227 P afde26b99c8d4e2e8ac019f9fa3fb40c [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:07.474749 25570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d61864f358a4a7a933222df4294b227" candidate_uuid: "afde26b99c8d4e2e8ac019f9fa3fb40c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39e676f08fe246fa9d7590ab40edd09e" is_pre_election: true
I20260620 18:38:07.475256 25570 raft_consensus.cc:2454] T 0d61864f358a4a7a933222df4294b227 P 39e676f08fe246fa9d7590ab40edd09e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate afde26b99c8d4e2e8ac019f9fa3fb40c in term 0.
I20260620 18:38:07.476960 25646 raft_consensus.cc:504] T 0d61864f358a4a7a933222df4294b227 P afde26b99c8d4e2e8ac019f9fa3fb40c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26f1de5d3e9744eebd1deb54b454f585" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 41579 } } peers { permanent_uuid: "afde26b99c8d4e2e8ac019f9fa3fb40c" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 35569 } } peers { permanent_uuid: "39e676f08fe246fa9d7590ab40edd09e" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 45969 } }
I20260620 18:38:07.477586 25646 leader_election.cc:288] T 0d61864f358a4a7a933222df4294b227 P afde26b99c8d4e2e8ac019f9fa3fb40c [CANDIDATE]: Term 1 election: Requested vote from peers 26f1de5d3e9744eebd1deb54b454f585 (127.23.115.1:41579), 39e676f08fe246fa9d7590ab40edd09e (127.23.115.3:45969)
I20260620 18:38:07.478048 25570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d61864f358a4a7a933222df4294b227" candidate_uuid: "afde26b99c8d4e2e8ac019f9fa3fb40c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39e676f08fe246fa9d7590ab40edd09e"
I20260620 18:38:07.478087 25292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d61864f358a4a7a933222df4294b227" candidate_uuid: "afde26b99c8d4e2e8ac019f9fa3fb40c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26f1de5d3e9744eebd1deb54b454f585"
I20260620 18:38:07.478259 25292 raft_consensus.cc:3037] T 0d61864f358a4a7a933222df4294b227 P 26f1de5d3e9744eebd1deb54b454f585 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:07.478262 25570 raft_consensus.cc:3037] T 0d61864f358a4a7a933222df4294b227 P 39e676f08fe246fa9d7590ab40edd09e [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:07.480429 25292 raft_consensus.cc:2454] T 0d61864f358a4a7a933222df4294b227 P 26f1de5d3e9744eebd1deb54b454f585 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate afde26b99c8d4e2e8ac019f9fa3fb40c in term 1.
I20260620 18:38:07.480859 25362 leader_election.cc:302] T 0d61864f358a4a7a933222df4294b227 P afde26b99c8d4e2e8ac019f9fa3fb40c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26f1de5d3e9744eebd1deb54b454f585, afde26b99c8d4e2e8ac019f9fa3fb40c; no voters:
I20260620 18:38:07.481210 25646 raft_consensus.cc:2781] T 0d61864f358a4a7a933222df4294b227 P afde26b99c8d4e2e8ac019f9fa3fb40c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:38:07.481762 25570 raft_consensus.cc:2454] T 0d61864f358a4a7a933222df4294b227 P 39e676f08fe246fa9d7590ab40edd09e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate afde26b99c8d4e2e8ac019f9fa3fb40c in term 1.
I20260620 18:38:07.482197 25646 raft_consensus.cc:686] T 0d61864f358a4a7a933222df4294b227 P afde26b99c8d4e2e8ac019f9fa3fb40c [term 1 LEADER]: Becoming Leader. State: Replica: afde26b99c8d4e2e8ac019f9fa3fb40c, State: Running, Role: LEADER
I20260620 18:38:07.482582 25646 consensus_queue.cc:237] T 0d61864f358a4a7a933222df4294b227 P afde26b99c8d4e2e8ac019f9fa3fb40c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26f1de5d3e9744eebd1deb54b454f585" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 41579 } } peers { permanent_uuid: "afde26b99c8d4e2e8ac019f9fa3fb40c" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 35569 } } peers { permanent_uuid: "39e676f08fe246fa9d7590ab40edd09e" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 45969 } }
I20260620 18:38:07.486011 25135 catalog_manager.cc:5495] T 0d61864f358a4a7a933222df4294b227 P afde26b99c8d4e2e8ac019f9fa3fb40c reported cstate change: term changed from 0 to 1, leader changed from <none> to afde26b99c8d4e2e8ac019f9fa3fb40c (127.23.115.2). New cstate: current_term: 1 leader_uuid: "afde26b99c8d4e2e8ac019f9fa3fb40c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26f1de5d3e9744eebd1deb54b454f585" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 41579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "afde26b99c8d4e2e8ac019f9fa3fb40c" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 35569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39e676f08fe246fa9d7590ab40edd09e" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 45969 } health_report { overall_health: UNKNOWN } } }
Jun 20 18:38:07 dist-test-slave-d645 krb5kdc[25090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980685, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.62@KRBTEST.COM for kudu/127.23.115.62@KRBTEST.COM
I20260620 18:38:07.546669 25230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:07.533366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50492 (local address 127.23.115.62:40053)
0620 18:38:07.533551 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:07.533558 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:07.534163 (+ 605us) server_negotiation.cc:407] Connection header received
0620 18:38:07.534336 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:07.534341 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:07.534419 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:07.534495 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:07.535685 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:07.536411 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:07.537671 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:07.538149 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:07.538249 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:07.541406 (+ 3157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:07.541411 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:07.541414 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:07.541441 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:07.544010 (+ 2569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:07.544736 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:07.544742 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:07.544745 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:07.544879 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:07.545237 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:07.545240 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:07.545243 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:07.545508 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:07.545632 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:07.546044 (+ 412us) server_negotiation.cc:299] Negotiation successful
0620 18:38:07.546328 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":103,"server-negotiator.queue_time_us":72}
I20260620 18:38:07.546669 25677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:07.533430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:40053 (local address 127.0.0.1:50492)
0620 18:38:07.534070 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:07.534088 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:07.534233 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:07.534535 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:07.534539 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:07.535280 (+ 741us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:07.535559 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:07.535568 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:07.536536 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:07.536540 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:07.537548 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:07.537559 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:07.538321 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:07.538327 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:07.538461 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:07.539073 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:07.539091 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:07.541253 (+ 2162us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:07.544161 (+ 2908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:07.544174 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:07.544183 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:07.544631 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:07.544977 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:07.544982 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:07.544986 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:07.545129 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:07.545626 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:07.545634 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:07.545803 (+ 169us) client_negotiation.cc:769] Sending connection context
0620 18:38:07.545952 (+ 149us) client_negotiation.cc:240] Negotiation successful
0620 18:38:07.546204 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":349,"threads_started":1}
I20260620 18:38:07.552054 25135 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.23.115.62@KRBTEST.COM'} at 127.0.0.1:50492:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 18:38:07.553162 25135 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:38:07.560079 25271 tablet_service.cc:1467] Processing CreateTablet for tablet bc1a1dfb4162428b94e7b053642e3817 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=986252b66644429091fc156c4bd6ecfa]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 18:38:07.560374 25550 tablet_service.cc:1467] Processing CreateTablet for tablet bc1a1dfb4162428b94e7b053642e3817 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=986252b66644429091fc156c4bd6ecfa]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 18:38:07.560633 25271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc1a1dfb4162428b94e7b053642e3817. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:07.560887 25550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc1a1dfb4162428b94e7b053642e3817. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:07.561477 25410 tablet_service.cc:1467] Processing CreateTablet for tablet bc1a1dfb4162428b94e7b053642e3817 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=986252b66644429091fc156c4bd6ecfa]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 18:38:07.561949 25410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc1a1dfb4162428b94e7b053642e3817. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:07.567646 25641 tablet_bootstrap.cc:492] T bc1a1dfb4162428b94e7b053642e3817 P 39e676f08fe246fa9d7590ab40edd09e: Bootstrap starting.
I20260620 18:38:07.568817 25642 tablet_bootstrap.cc:492] T bc1a1dfb4162428b94e7b053642e3817 P afde26b99c8d4e2e8ac019f9fa3fb40c: Bootstrap starting.
I20260620 18:38:07.569042 25639 tablet_bootstrap.cc:492] T bc1a1dfb4162428b94e7b053642e3817 P 26f1de5d3e9744eebd1deb54b454f585: Bootstrap starting.
I20260620 18:38:07.569711 25641 tablet_bootstrap.cc:654] T bc1a1dfb4162428b94e7b053642e3817 P 39e676f08fe246fa9d7590ab40edd09e: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:07.570346 25642 tablet_bootstrap.cc:654] T bc1a1dfb4162428b94e7b053642e3817 P afde26b99c8d4e2e8ac019f9fa3fb40c: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:07.570611 25639 tablet_bootstrap.cc:654] T bc1a1dfb4162428b94e7b053642e3817 P 26f1de5d3e9744eebd1deb54b454f585: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:07.581121 25642 tablet_bootstrap.cc:492] T bc1a1dfb4162428b94e7b053642e3817 P afde26b99c8d4e2e8ac019f9fa3fb40c: No bootstrap required, opened a new log
I20260620 18:38:07.581358 25642 ts_tablet_manager.cc:1397] T bc1a1dfb4162428b94e7b053642e3817 P afde26b99c8d4e2e8ac019f9fa3fb40c: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.004s
I20260620 18:38:07.581729 25641 tablet_bootstrap.cc:492] T bc1a1dfb4162428b94e7b053642e3817 P 39e676f08fe246fa9d7590ab40edd09e: No bootstrap required, opened a new log
I20260620 18:38:07.581921 25641 ts_tablet_manager.cc:1397] T bc1a1dfb4162428b94e7b053642e3817 P 39e676f08fe246fa9d7590ab40edd09e: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.005s
I20260620 18:38:07.582283 25642 raft_consensus.cc:348] T bc1a1dfb4162428b94e7b053642e3817 P afde26b99c8d4e2e8ac019f9fa3fb40c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26f1de5d3e9744eebd1deb54b454f585" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 41579 } } peers { permanent_uuid: "afde26b99c8d4e2e8ac019f9fa3fb40c" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 35569 } } peers { permanent_uuid: "39e676f08fe246fa9d7590ab40edd09e" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 45969 } }
I20260620 18:38:07.582592 25642 raft_consensus.cc:374] T bc1a1dfb4162428b94e7b053642e3817 P afde26b99c8d4e2e8ac019f9fa3fb40c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:07.582767 25642 raft_consensus.cc:729] T bc1a1dfb4162428b94e7b053642e3817 P afde26b99c8d4e2e8ac019f9fa3fb40c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afde26b99c8d4e2e8ac019f9fa3fb40c, State: Initialized, Role: FOLLOWER
I20260620 18:38:07.582717 25641 raft_consensus.cc:348] T bc1a1dfb4162428b94e7b053642e3817 P 39e676f08fe246fa9d7590ab40edd09e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26f1de5d3e9744eebd1deb54b454f585" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 41579 } } peers { permanent_uuid: "afde26b99c8d4e2e8ac019f9fa3fb40c" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 35569 } } peers { permanent_uuid: "39e676f08fe246fa9d7590ab40edd09e" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 45969 } }
I20260620 18:38:07.582981 25641 raft_consensus.cc:374] T bc1a1dfb4162428b94e7b053642e3817 P 39e676f08fe246fa9d7590ab40edd09e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:07.583072 25641 raft_consensus.cc:729] T bc1a1dfb4162428b94e7b053642e3817 P 39e676f08fe246fa9d7590ab40edd09e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39e676f08fe246fa9d7590ab40edd09e, State: Initialized, Role: FOLLOWER
I20260620 18:38:07.583366 25641 consensus_queue.cc:260] T bc1a1dfb4162428b94e7b053642e3817 P 39e676f08fe246fa9d7590ab40edd09e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26f1de5d3e9744eebd1deb54b454f585" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 41579 } } peers { permanent_uuid: "afde26b99c8d4e2e8ac019f9fa3fb40c" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 35569 } } peers { permanent_uuid: "39e676f08fe246fa9d7590ab40edd09e" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 45969 } }
I20260620 18:38:07.583366 25642 consensus_queue.cc:260] T bc1a1dfb4162428b94e7b053642e3817 P afde26b99c8d4e2e8ac019f9fa3fb40c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26f1de5d3e9744eebd1deb54b454f585" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 41579 } } peers { permanent_uuid: "afde26b99c8d4e2e8ac019f9fa3fb40c" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 35569 } } peers { permanent_uuid: "39e676f08fe246fa9d7590ab40edd09e" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 45969 } }
I20260620 18:38:07.583930 25641 ts_tablet_manager.cc:1428] T bc1a1dfb4162428b94e7b053642e3817 P 39e676f08fe246fa9d7590ab40edd09e: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 18:38:07.583894 25646 tablet_replica.cc:439] T bc1a1dfb4162428b94e7b053642e3817 P afde26b99c8d4e2e8ac019f9fa3fb40c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26f1de5d3e9744eebd1deb54b454f585" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 41579 } } peers { permanent_uuid: "afde26b99c8d4e2e8ac019f9fa3fb40c" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 35569 } } peers { permanent_uuid: "39e676f08fe246fa9d7590ab40edd09e" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 45969 } } }
I20260620 18:38:07.584148 25646 tablet_replica.cc:442] T bc1a1dfb4162428b94e7b053642e3817 P afde26b99c8d4e2e8ac019f9fa3fb40c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:38:07.585248 25642 ts_tablet_manager.cc:1428] T bc1a1dfb4162428b94e7b053642e3817 P afde26b99c8d4e2e8ac019f9fa3fb40c: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260620 18:38:07.585685 25647 tablet_replica.cc:439] T bc1a1dfb4162428b94e7b053642e3817 P 39e676f08fe246fa9d7590ab40edd09e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26f1de5d3e9744eebd1deb54b454f585" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 41579 } } peers { permanent_uuid: "afde26b99c8d4e2e8ac019f9fa3fb40c" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 35569 } } peers { permanent_uuid: "39e676f08fe246fa9d7590ab40edd09e" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 45969 } } }
I20260620 18:38:07.585940 25647 tablet_replica.cc:442] T bc1a1dfb4162428b94e7b053642e3817 P 39e676f08fe246fa9d7590ab40edd09e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:38:07.588452 25639 tablet_bootstrap.cc:492] T bc1a1dfb4162428b94e7b053642e3817 P 26f1de5d3e9744eebd1deb54b454f585: No bootstrap required, opened a new log
I20260620 18:38:07.588677 25639 ts_tablet_manager.cc:1397] T bc1a1dfb4162428b94e7b053642e3817 P 26f1de5d3e9744eebd1deb54b454f585: Time spent bootstrapping tablet: real 0.020s user 0.007s sys 0.004s
I20260620 18:38:07.589438 25639 raft_consensus.cc:348] T bc1a1dfb4162428b94e7b053642e3817 P 26f1de5d3e9744eebd1deb54b454f585 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26f1de5d3e9744eebd1deb54b454f585" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 41579 } } peers { permanent_uuid: "afde26b99c8d4e2e8ac019f9fa3fb40c" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 35569 } } peers { permanent_uuid: "39e676f08fe246fa9d7590ab40edd09e" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 45969 } }
I20260620 18:38:07.589740 25639 raft_consensus.cc:374] T bc1a1dfb4162428b94e7b053642e3817 P 26f1de5d3e9744eebd1deb54b454f585 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:07.589849 25639 raft_consensus.cc:729] T bc1a1dfb4162428b94e7b053642e3817 P 26f1de5d3e9744eebd1deb54b454f585 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26f1de5d3e9744eebd1deb54b454f585, State: Initialized, Role: FOLLOWER
I20260620 18:38:07.590162 25639 consensus_queue.cc:260] T bc1a1dfb4162428b94e7b053642e3817 P 26f1de5d3e9744eebd1deb54b454f585 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26f1de5d3e9744eebd1deb54b454f585" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 41579 } } peers { permanent_uuid: "afde26b99c8d4e2e8ac019f9fa3fb40c" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 35569 } } peers { permanent_uuid: "39e676f08fe246fa9d7590ab40edd09e" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 45969 } }
I20260620 18:38:07.590821 25639 ts_tablet_manager.cc:1428] T bc1a1dfb4162428b94e7b053642e3817 P 26f1de5d3e9744eebd1deb54b454f585: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260620 18:38:07.590698 25645 tablet_replica.cc:439] T bc1a1dfb4162428b94e7b053642e3817 P 26f1de5d3e9744eebd1deb54b454f585: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26f1de5d3e9744eebd1deb54b454f585" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 41579 } } peers { permanent_uuid: "afde26b99c8d4e2e8ac019f9fa3fb40c" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 35569 } } peers { permanent_uuid: "39e676f08fe246fa9d7590ab40edd09e" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 45969 } } }
I20260620 18:38:07.591089 25645 tablet_replica.cc:442] T bc1a1dfb4162428b94e7b053642e3817 P 26f1de5d3e9744eebd1deb54b454f585: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:38:07.768365 25647 raft_consensus.cc:482] T bc1a1dfb4162428b94e7b053642e3817 P 39e676f08fe246fa9d7590ab40edd09e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:38:07.768570 25647 raft_consensus.cc:504] T bc1a1dfb4162428b94e7b053642e3817 P 39e676f08fe246fa9d7590ab40edd09e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26f1de5d3e9744eebd1deb54b454f585" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 41579 } } peers { permanent_uuid: "afde26b99c8d4e2e8ac019f9fa3fb40c" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 35569 } } peers { permanent_uuid: "39e676f08fe246fa9d7590ab40edd09e" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 45969 } }
I20260620 18:38:07.769299 25647 leader_election.cc:288] T bc1a1dfb4162428b94e7b053642e3817 P 39e676f08fe246fa9d7590ab40edd09e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26f1de5d3e9744eebd1deb54b454f585 (127.23.115.1:41579), afde26b99c8d4e2e8ac019f9fa3fb40c (127.23.115.2:35569)
I20260620 18:38:07.769883 25292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc1a1dfb4162428b94e7b053642e3817" candidate_uuid: "39e676f08fe246fa9d7590ab40edd09e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26f1de5d3e9744eebd1deb54b454f585" is_pre_election: true
I20260620 18:38:07.769891 25436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc1a1dfb4162428b94e7b053642e3817" candidate_uuid: "39e676f08fe246fa9d7590ab40edd09e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "afde26b99c8d4e2e8ac019f9fa3fb40c" is_pre_election: true
I20260620 18:38:07.770133 25436 raft_consensus.cc:2454] T bc1a1dfb4162428b94e7b053642e3817 P afde26b99c8d4e2e8ac019f9fa3fb40c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39e676f08fe246fa9d7590ab40edd09e in term 0.
I20260620 18:38:07.770145 25292 raft_consensus.cc:2454] T bc1a1dfb4162428b94e7b053642e3817 P 26f1de5d3e9744eebd1deb54b454f585 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39e676f08fe246fa9d7590ab40edd09e in term 0.
I20260620 18:38:07.770541 25501 leader_election.cc:302] T bc1a1dfb4162428b94e7b053642e3817 P 39e676f08fe246fa9d7590ab40edd09e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26f1de5d3e9744eebd1deb54b454f585, 39e676f08fe246fa9d7590ab40edd09e; no voters:
I20260620 18:38:07.770902 25647 raft_consensus.cc:2781] T bc1a1dfb4162428b94e7b053642e3817 P 39e676f08fe246fa9d7590ab40edd09e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:38:07.771057 25647 raft_consensus.cc:482] T bc1a1dfb4162428b94e7b053642e3817 P 39e676f08fe246fa9d7590ab40edd09e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:38:07.771150 25647 raft_consensus.cc:3037] T bc1a1dfb4162428b94e7b053642e3817 P 39e676f08fe246fa9d7590ab40edd09e [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:07.773537 25647 raft_consensus.cc:504] T bc1a1dfb4162428b94e7b053642e3817 P 39e676f08fe246fa9d7590ab40edd09e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26f1de5d3e9744eebd1deb54b454f585" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 41579 } } peers { permanent_uuid: "afde26b99c8d4e2e8ac019f9fa3fb40c" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 35569 } } peers { permanent_uuid: "39e676f08fe246fa9d7590ab40edd09e" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 45969 } }
I20260620 18:38:07.774144 25647 leader_election.cc:288] T bc1a1dfb4162428b94e7b053642e3817 P 39e676f08fe246fa9d7590ab40edd09e [CANDIDATE]: Term 1 election: Requested vote from peers 26f1de5d3e9744eebd1deb54b454f585 (127.23.115.1:41579), afde26b99c8d4e2e8ac019f9fa3fb40c (127.23.115.2:35569)
I20260620 18:38:07.774580 25292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc1a1dfb4162428b94e7b053642e3817" candidate_uuid: "39e676f08fe246fa9d7590ab40edd09e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26f1de5d3e9744eebd1deb54b454f585"
I20260620 18:38:07.774622 25436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc1a1dfb4162428b94e7b053642e3817" candidate_uuid: "39e676f08fe246fa9d7590ab40edd09e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "afde26b99c8d4e2e8ac019f9fa3fb40c"
I20260620 18:38:07.774825 25436 raft_consensus.cc:3037] T bc1a1dfb4162428b94e7b053642e3817 P afde26b99c8d4e2e8ac019f9fa3fb40c [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:07.774842 25292 raft_consensus.cc:3037] T bc1a1dfb4162428b94e7b053642e3817 P 26f1de5d3e9744eebd1deb54b454f585 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:07.776887 25436 raft_consensus.cc:2454] T bc1a1dfb4162428b94e7b053642e3817 P afde26b99c8d4e2e8ac019f9fa3fb40c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39e676f08fe246fa9d7590ab40edd09e in term 1.
I20260620 18:38:07.777302 25502 leader_election.cc:302] T bc1a1dfb4162428b94e7b053642e3817 P 39e676f08fe246fa9d7590ab40edd09e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39e676f08fe246fa9d7590ab40edd09e, afde26b99c8d4e2e8ac019f9fa3fb40c; no voters:
I20260620 18:38:07.777616 25647 raft_consensus.cc:2781] T bc1a1dfb4162428b94e7b053642e3817 P 39e676f08fe246fa9d7590ab40edd09e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:38:07.777774 25647 raft_consensus.cc:686] T bc1a1dfb4162428b94e7b053642e3817 P 39e676f08fe246fa9d7590ab40edd09e [term 1 LEADER]: Becoming Leader. State: Replica: 39e676f08fe246fa9d7590ab40edd09e, State: Running, Role: LEADER
I20260620 18:38:07.777804 25292 raft_consensus.cc:2454] T bc1a1dfb4162428b94e7b053642e3817 P 26f1de5d3e9744eebd1deb54b454f585 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39e676f08fe246fa9d7590ab40edd09e in term 1.
I20260620 18:38:07.778064 25647 consensus_queue.cc:237] T bc1a1dfb4162428b94e7b053642e3817 P 39e676f08fe246fa9d7590ab40edd09e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26f1de5d3e9744eebd1deb54b454f585" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 41579 } } peers { permanent_uuid: "afde26b99c8d4e2e8ac019f9fa3fb40c" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 35569 } } peers { permanent_uuid: "39e676f08fe246fa9d7590ab40edd09e" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 45969 } }
I20260620 18:38:07.779784 25649 tablet_replica.cc:439] T bc1a1dfb4162428b94e7b053642e3817 P 39e676f08fe246fa9d7590ab40edd09e: TxnStatusTablet state changed. Reason: New leader 39e676f08fe246fa9d7590ab40edd09e. Latest consensus state: current_term: 1 leader_uuid: "39e676f08fe246fa9d7590ab40edd09e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26f1de5d3e9744eebd1deb54b454f585" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 41579 } } peers { permanent_uuid: "afde26b99c8d4e2e8ac019f9fa3fb40c" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 35569 } } peers { permanent_uuid: "39e676f08fe246fa9d7590ab40edd09e" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 45969 } } }
I20260620 18:38:07.780114 25649 tablet_replica.cc:442] T bc1a1dfb4162428b94e7b053642e3817 P 39e676f08fe246fa9d7590ab40edd09e: This TxnStatusTablet replica's current role is: LEADER
I20260620 18:38:07.780661 25682 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 18:38:07.780869 25682 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 18:38:07.781985 25135 catalog_manager.cc:5495] T bc1a1dfb4162428b94e7b053642e3817 P 39e676f08fe246fa9d7590ab40edd09e reported cstate change: term changed from 0 to 1, leader changed from <none> to 39e676f08fe246fa9d7590ab40edd09e (127.23.115.3). New cstate: current_term: 1 leader_uuid: "39e676f08fe246fa9d7590ab40edd09e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26f1de5d3e9744eebd1deb54b454f585" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 41579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "afde26b99c8d4e2e8ac019f9fa3fb40c" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 35569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39e676f08fe246fa9d7590ab40edd09e" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 45969 } health_report { overall_health: HEALTHY } } }
I20260620 18:38:07.826148 25638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:07.820719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50350 (local address 127.23.115.3:45969)
0620 18:38:07.821213 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:07.821217 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:07.821234 (+ 17us) server_negotiation.cc:407] Connection header received
0620 18:38:07.821306 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:07.821310 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:07.821382 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:07.821487 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:38:07.821904 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:07.822691 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:07.824496 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:07.824937 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:07.825043 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:07.825560 (+ 517us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:38:07.825659 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:38:07.825725 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:07.825874 (+ 149us) server_negotiation.cc:299] Negotiation successful
0620 18:38:07.825932 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390}
I20260620 18:38:07.826318 25677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:07.820620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.3:45969 (local address 127.0.0.1:50350)
0620 18:38:07.820856 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:07.820873 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:07.820981 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:07.821555 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:07.821561 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:07.821575 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 18:38:07.821768 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:07.821778 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:07.822824 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:07.822831 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:07.824346 (+ 1515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:07.824361 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:07.825096 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:07.825103 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:07.825300 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:07.825370 (+ 70us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 18:38:07.825752 (+ 382us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 18:38:07.825772 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 18:38:07.825849 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 18:38:07.826009 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260620 18:38:07.829777 25292 raft_consensus.cc:1261] T bc1a1dfb4162428b94e7b053642e3817 P 26f1de5d3e9744eebd1deb54b454f585 [term 1 FOLLOWER]: Refusing update from remote peer 39e676f08fe246fa9d7590ab40edd09e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:38:07.829780 25436 raft_consensus.cc:1261] T bc1a1dfb4162428b94e7b053642e3817 P afde26b99c8d4e2e8ac019f9fa3fb40c [term 1 FOLLOWER]: Refusing update from remote peer 39e676f08fe246fa9d7590ab40edd09e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:38:07.830380 25649 consensus_queue.cc:1035] T bc1a1dfb4162428b94e7b053642e3817 P 39e676f08fe246fa9d7590ab40edd09e [LEADER]: Connected to new peer: Peer: permanent_uuid: "26f1de5d3e9744eebd1deb54b454f585" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 41579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:38:07.830624 25647 consensus_queue.cc:1035] T bc1a1dfb4162428b94e7b053642e3817 P 39e676f08fe246fa9d7590ab40edd09e [LEADER]: Connected to new peer: Peer: permanent_uuid: "afde26b99c8d4e2e8ac019f9fa3fb40c" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 35569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:38:07.835770 25645 tablet_replica.cc:439] T bc1a1dfb4162428b94e7b053642e3817 P 26f1de5d3e9744eebd1deb54b454f585: TxnStatusTablet state changed. Reason: New leader 39e676f08fe246fa9d7590ab40edd09e. Latest consensus state: current_term: 1 leader_uuid: "39e676f08fe246fa9d7590ab40edd09e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26f1de5d3e9744eebd1deb54b454f585" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 41579 } } peers { permanent_uuid: "afde26b99c8d4e2e8ac019f9fa3fb40c" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 35569 } } peers { permanent_uuid: "39e676f08fe246fa9d7590ab40edd09e" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 45969 } } }
I20260620 18:38:07.836115 25645 tablet_replica.cc:442] T bc1a1dfb4162428b94e7b053642e3817 P 26f1de5d3e9744eebd1deb54b454f585: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:38:07.838193 25647 tablet_replica.cc:439] T bc1a1dfb4162428b94e7b053642e3817 P 39e676f08fe246fa9d7590ab40edd09e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "39e676f08fe246fa9d7590ab40edd09e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26f1de5d3e9744eebd1deb54b454f585" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 41579 } } peers { permanent_uuid: "afde26b99c8d4e2e8ac019f9fa3fb40c" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 35569 } } peers { permanent_uuid: "39e676f08fe246fa9d7590ab40edd09e" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 45969 } } }
I20260620 18:38:07.838573 25647 tablet_replica.cc:442] T bc1a1dfb4162428b94e7b053642e3817 P 39e676f08fe246fa9d7590ab40edd09e: This TxnStatusTablet replica's current role is: LEADER
I20260620 18:38:07.839175 25652 mvcc.cc:205] Tried to move back new op lower bound from 7298992897345302528 to 7298992897141514240. Current Snapshot: MvccSnapshot[applied={T|T < 7298992897345302528 or (T in {7298992897345302528})}]
I20260620 18:38:07.840526 25646 tablet_replica.cc:439] T bc1a1dfb4162428b94e7b053642e3817 P afde26b99c8d4e2e8ac019f9fa3fb40c: TxnStatusTablet state changed. Reason: New leader 39e676f08fe246fa9d7590ab40edd09e. Latest consensus state: current_term: 1 leader_uuid: "39e676f08fe246fa9d7590ab40edd09e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26f1de5d3e9744eebd1deb54b454f585" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 41579 } } peers { permanent_uuid: "afde26b99c8d4e2e8ac019f9fa3fb40c" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 35569 } } peers { permanent_uuid: "39e676f08fe246fa9d7590ab40edd09e" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 45969 } } }
I20260620 18:38:07.840790 25646 tablet_replica.cc:442] T bc1a1dfb4162428b94e7b053642e3817 P afde26b99c8d4e2e8ac019f9fa3fb40c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:38:07.841300 25654 mvcc.cc:205] Tried to move back new op lower bound from 7298992897345302528 to 7298992897141514240. Current Snapshot: MvccSnapshot[applied={T|T < 7298992897345302528}]
I20260620 18:38:07.841581 25645 tablet_replica.cc:439] T bc1a1dfb4162428b94e7b053642e3817 P 26f1de5d3e9744eebd1deb54b454f585: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "39e676f08fe246fa9d7590ab40edd09e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26f1de5d3e9744eebd1deb54b454f585" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 41579 } } peers { permanent_uuid: "afde26b99c8d4e2e8ac019f9fa3fb40c" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 35569 } } peers { permanent_uuid: "39e676f08fe246fa9d7590ab40edd09e" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 45969 } } }
I20260620 18:38:07.841789 25645 tablet_replica.cc:442] T bc1a1dfb4162428b94e7b053642e3817 P 26f1de5d3e9744eebd1deb54b454f585: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:38:07.842446 25657 tablet_replica.cc:439] T bc1a1dfb4162428b94e7b053642e3817 P 39e676f08fe246fa9d7590ab40edd09e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "39e676f08fe246fa9d7590ab40edd09e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26f1de5d3e9744eebd1deb54b454f585" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 41579 } } peers { permanent_uuid: "afde26b99c8d4e2e8ac019f9fa3fb40c" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 35569 } } peers { permanent_uuid: "39e676f08fe246fa9d7590ab40edd09e" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 45969 } } }
I20260620 18:38:07.842809 25657 tablet_replica.cc:442] T bc1a1dfb4162428b94e7b053642e3817 P 39e676f08fe246fa9d7590ab40edd09e: This TxnStatusTablet replica's current role is: LEADER
I20260620 18:38:07.844111 25646 tablet_replica.cc:439] T bc1a1dfb4162428b94e7b053642e3817 P afde26b99c8d4e2e8ac019f9fa3fb40c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "39e676f08fe246fa9d7590ab40edd09e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26f1de5d3e9744eebd1deb54b454f585" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 41579 } } peers { permanent_uuid: "afde26b99c8d4e2e8ac019f9fa3fb40c" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 35569 } } peers { permanent_uuid: "39e676f08fe246fa9d7590ab40edd09e" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 45969 } } }
I20260620 18:38:07.844373 25646 tablet_replica.cc:442] T bc1a1dfb4162428b94e7b053642e3817 P afde26b99c8d4e2e8ac019f9fa3fb40c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:38:07.858461 25687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:07.851733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60888 (local address 127.23.115.2:35569)
0620 18:38:07.852616 (+ 883us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:07.852621 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:07.852642 (+ 21us) server_negotiation.cc:407] Connection header received
0620 18:38:07.852726 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:07.852731 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:07.852814 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:07.852963 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:38:07.853338 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:07.854517 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:07.856079 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:07.856558 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:07.856683 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:07.857212 (+ 529us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:38:07.857780 (+ 568us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:38:07.857938 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:07.858034 (+ 96us) server_negotiation.cc:299] Negotiation successful
0620 18:38:07.858151 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":765,"thread_start_us":642,"threads_started":1}
I20260620 18:38:07.874011 25690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:07.867882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.3:45969 (local address 127.23.115.2:42029)
0620 18:38:07.868407 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:07.868423 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:07.868534 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:07.868874 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:07.868877 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:07.868887 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 18:38:07.869146 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:07.869154 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:07.870597 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:07.870601 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:07.871733 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:07.871746 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:07.872853 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:07.872858 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:07.872985 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:07.873036 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 18:38:07.873489 (+ 453us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 18:38:07.873497 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 18:38:07.873560 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 18:38:07.873732 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":289,"threads_started":1}
I20260620 18:38:07.874009 25638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:07.867818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.2:42029 (local address 127.23.115.3:45969)
0620 18:38:07.867988 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:07.867993 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:07.868500 (+ 507us) server_negotiation.cc:407] Connection header received
0620 18:38:07.868647 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:07.868653 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:07.868739 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:07.868832 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:38:07.869285 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:07.870456 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:07.871869 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:07.872649 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:07.872783 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:07.873231 (+ 448us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:38:07.873393 (+ 162us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:38:07.873485 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:07.873597 (+ 112us) server_negotiation.cc:299] Negotiation successful
0620 18:38:07.873678 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 18:38:07.884881 25291 raft_consensus.cc:1261] T 0d61864f358a4a7a933222df4294b227 P 26f1de5d3e9744eebd1deb54b454f585 [term 1 FOLLOWER]: Refusing update from remote peer afde26b99c8d4e2e8ac019f9fa3fb40c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:38:07.885298 25570 raft_consensus.cc:1261] T 0d61864f358a4a7a933222df4294b227 P 39e676f08fe246fa9d7590ab40edd09e [term 1 FOLLOWER]: Refusing update from remote peer afde26b99c8d4e2e8ac019f9fa3fb40c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:38:07.885610 25670 consensus_queue.cc:1035] T 0d61864f358a4a7a933222df4294b227 P afde26b99c8d4e2e8ac019f9fa3fb40c [LEADER]: Connected to new peer: Peer: permanent_uuid: "26f1de5d3e9744eebd1deb54b454f585" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 41579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:38:07.886018 25646 consensus_queue.cc:1035] T 0d61864f358a4a7a933222df4294b227 P afde26b99c8d4e2e8ac019f9fa3fb40c [LEADER]: Connected to new peer: Peer: permanent_uuid: "39e676f08fe246fa9d7590ab40edd09e" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 45969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:38:07.893504 25653 mvcc.cc:205] Tried to move back new op lower bound from 7298992897570742272 to 7298992895932145664. Current Snapshot: MvccSnapshot[applied={T|T < 7298992897570742272}]
I20260620 18:38:07.924178 25687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:07.917453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.3:39509 (local address 127.23.115.2:35569)
0620 18:38:07.917621 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:07.917627 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:07.918135 (+ 508us) server_negotiation.cc:407] Connection header received
0620 18:38:07.918397 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:07.918402 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:07.918484 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:07.918563 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:38:07.919177 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:07.920313 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:07.921979 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:07.922636 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:07.922764 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:07.923402 (+ 638us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:38:07.923547 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:38:07.923635 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:07.923807 (+ 172us) server_negotiation.cc:299] Negotiation successful
0620 18:38:07.923890 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 18:38:07.924239 25695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:07.917370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.2:35569 (local address 127.23.115.3:39509)
0620 18:38:07.918037 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:07.918057 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:07.918213 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:07.918594 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:07.918599 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:07.918616 (+ 17us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 18:38:07.919019 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:07.919034 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:07.920486 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:07.920490 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:07.921839 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:07.921852 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:07.922969 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:07.922976 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:07.923175 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:07.923238 (+ 63us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 18:38:07.923637 (+ 399us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 18:38:07.923648 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 18:38:07.923732 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 18:38:07.923866 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":314,"threads_started":1}
I20260620 18:38:07.960894 25134 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56558:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:38:07.961146 25134 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56558:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:38:07.964751 25134 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d9e48bfe423541ea965d5203d2ce8dbe: Sending DeleteTablet for 3 replicas of tablet 0d61864f358a4a7a933222df4294b227
I20260620 18:38:07.965492 25271 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d61864f358a4a7a933222df4294b227 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:38:07 UTC) from {username='kudu', principal='kudu/127.23.115.62@KRBTEST.COM'} at 127.0.0.1:37678
I20260620 18:38:07.965591 25410 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d61864f358a4a7a933222df4294b227 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:38:07 UTC) from {username='kudu', principal='kudu/127.23.115.62@KRBTEST.COM'} at 127.0.0.1:49194
I20260620 18:38:07.969339 25550 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d61864f358a4a7a933222df4294b227 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:38:07 UTC) from {username='kudu', principal='kudu/127.23.115.62@KRBTEST.COM'} at 127.0.0.1:47942
I20260620 18:38:07.978420 25698 tablet_replica.cc:330] T 0d61864f358a4a7a933222df4294b227 P 26f1de5d3e9744eebd1deb54b454f585: stopping tablet replica
I20260620 18:38:07.978696 25698 raft_consensus.cc:2229] T 0d61864f358a4a7a933222df4294b227 P 26f1de5d3e9744eebd1deb54b454f585 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:38:07.979009 25698 raft_consensus.cc:2258] T 0d61864f358a4a7a933222df4294b227 P 26f1de5d3e9744eebd1deb54b454f585 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:38:07.980201 25698 ts_tablet_manager.cc:1905] T 0d61864f358a4a7a933222df4294b227 P 26f1de5d3e9744eebd1deb54b454f585: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:38:07.987749 25698 ts_tablet_manager.cc:1918] T 0d61864f358a4a7a933222df4294b227 P 26f1de5d3e9744eebd1deb54b454f585: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:38:07.987972 25698 log.cc:1196] T 0d61864f358a4a7a933222df4294b227 P 26f1de5d3e9744eebd1deb54b454f585: Deleting WAL directory at /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980676253068-24012-0/minicluster-data/ts-0/wal/wals/0d61864f358a4a7a933222df4294b227
I20260620 18:38:07.987974 25699 tablet_replica.cc:330] T 0d61864f358a4a7a933222df4294b227 P afde26b99c8d4e2e8ac019f9fa3fb40c: stopping tablet replica
I20260620 18:38:07.988219 25699 raft_consensus.cc:2229] T 0d61864f358a4a7a933222df4294b227 P afde26b99c8d4e2e8ac019f9fa3fb40c [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:38:07.988580 25698 ts_tablet_manager.cc:1939] T 0d61864f358a4a7a933222df4294b227 P 26f1de5d3e9744eebd1deb54b454f585: Deleting consensus metadata
I20260620 18:38:07.988790 25699 raft_consensus.cc:2258] T 0d61864f358a4a7a933222df4294b227 P afde26b99c8d4e2e8ac019f9fa3fb40c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:38:07.989957 25699 ts_tablet_manager.cc:1905] T 0d61864f358a4a7a933222df4294b227 P afde26b99c8d4e2e8ac019f9fa3fb40c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:38:07.995428 25119 catalog_manager.cc:4841] TS 26f1de5d3e9744eebd1deb54b454f585 (127.23.115.1:41579): tablet 0d61864f358a4a7a933222df4294b227 (table test-table [id=ad7b132f700f4d1eb9b309454345e5c3]) successfully deleted
I20260620 18:38:07.997123 25699 ts_tablet_manager.cc:1918] T 0d61864f358a4a7a933222df4294b227 P afde26b99c8d4e2e8ac019f9fa3fb40c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:38:07.997287 25699 log.cc:1196] T 0d61864f358a4a7a933222df4294b227 P afde26b99c8d4e2e8ac019f9fa3fb40c: Deleting WAL directory at /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980676253068-24012-0/minicluster-data/ts-1/wal/wals/0d61864f358a4a7a933222df4294b227
I20260620 18:38:07.997714 25699 ts_tablet_manager.cc:1939] T 0d61864f358a4a7a933222df4294b227 P afde26b99c8d4e2e8ac019f9fa3fb40c: Deleting consensus metadata
I20260620 18:38:07.998973 25120 catalog_manager.cc:4841] TS afde26b99c8d4e2e8ac019f9fa3fb40c (127.23.115.2:35569): tablet 0d61864f358a4a7a933222df4294b227 (table test-table [id=ad7b132f700f4d1eb9b309454345e5c3]) successfully deleted
I20260620 18:38:07.997745 25700 tablet_replica.cc:330] T 0d61864f358a4a7a933222df4294b227 P 39e676f08fe246fa9d7590ab40edd09e: stopping tablet replica
I20260620 18:38:07.999377 25700 raft_consensus.cc:2229] T 0d61864f358a4a7a933222df4294b227 P 39e676f08fe246fa9d7590ab40edd09e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:38:07.999655 25700 raft_consensus.cc:2258] T 0d61864f358a4a7a933222df4294b227 P 39e676f08fe246fa9d7590ab40edd09e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:38:08.000849 25700 ts_tablet_manager.cc:1905] T 0d61864f358a4a7a933222df4294b227 P 39e676f08fe246fa9d7590ab40edd09e: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 20 18:38:08 dist-test-slave-d645 krb5kdc[25090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980687, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.115.1@KRBTEST.COM
I20260620 18:38:08.008881 25700 ts_tablet_manager.cc:1918] T 0d61864f358a4a7a933222df4294b227 P 39e676f08fe246fa9d7590ab40edd09e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:38:08.009114 25700 log.cc:1196] T 0d61864f358a4a7a933222df4294b227 P 39e676f08fe246fa9d7590ab40edd09e: Deleting WAL directory at /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980676253068-24012-0/minicluster-data/ts-2/wal/wals/0d61864f358a4a7a933222df4294b227
I20260620 18:38:08.009717 25700 ts_tablet_manager.cc:1939] T 0d61864f358a4a7a933222df4294b227 P 39e676f08fe246fa9d7590ab40edd09e: Deleting consensus metadata
I20260620 18:38:08.010902 25119 catalog_manager.cc:4841] TS 39e676f08fe246fa9d7590ab40edd09e (127.23.115.3:45969): tablet 0d61864f358a4a7a933222df4294b227 (table test-table [id=ad7b132f700f4d1eb9b309454345e5c3]) successfully deleted
I20260620 18:38:08.016662 25702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:07.985664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33698 (local address 127.23.115.1:41579)
0620 18:38:07.995357 (+ 9693us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:07.995363 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:07.998318 (+ 2955us) server_negotiation.cc:407] Connection header received
0620 18:38:07.998387 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:07.998392 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:07.998484 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:07.998579 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:08.000569 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:08.001635 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:08.003762 (+ 2127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:08.004374 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:08.004563 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:08.009554 (+ 4991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:08.009572 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:08.009592 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:08.009656 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:08.012901 (+ 3245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:08.013923 (+ 1022us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:08.013938 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:08.013941 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:08.014250 (+ 309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:08.014959 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:08.014965 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:08.014969 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:08.015380 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:08.015658 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:08.016051 (+ 393us) server_negotiation.cc:299] Negotiation successful
0620 18:38:08.016299 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9553,"thread_start_us":9448,"threads_started":1}
W20260620 18:38:08.017817 25337 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33698
I20260620 18:38:08.035727 25230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:08.021184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50500 (local address 127.23.115.62:40053)
0620 18:38:08.021409 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:08.021416 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:08.021829 (+ 413us) server_negotiation.cc:407] Connection header received
0620 18:38:08.021984 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:08.021990 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:08.022059 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:08.022139 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:08.023665 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:08.024765 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:08.026009 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:08.026662 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:08.026818 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:08.029231 (+ 2413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:08.029237 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:08.029241 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:08.029272 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:08.031714 (+ 2442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:08.032451 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:08.032460 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:08.032464 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:08.032656 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:08.033341 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:08.033347 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:08.033350 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:08.033865 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:08.034219 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:08.035166 (+ 947us) server_negotiation.cc:299] Negotiation successful
0620 18:38:08.035466 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260620 18:38:08.036845 25134 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50500
I20260620 18:38:08.038715 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 25206
I20260620 18:38:08.052918 25702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:08.039862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33700 (local address 127.23.115.1:41579)
0620 18:38:08.040054 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:08.040060 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:08.041770 (+ 1710us) server_negotiation.cc:407] Connection header received
0620 18:38:08.041915 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:08.041920 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:08.041983 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:08.042069 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:08.043209 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:08.043886 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:08.044999 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:08.045411 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:08.045514 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:08.047665 (+ 2151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:08.047670 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:08.047673 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:08.047704 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:08.050254 (+ 2550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:08.050987 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:08.050993 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:08.050997 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:08.051131 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:08.051466 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:08.051471 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:08.051475 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:08.051748 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:08.051971 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:08.052249 (+ 278us) server_negotiation.cc:299] Negotiation successful
0620 18:38:08.052461 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260620 18:38:08.053545 25337 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33700
W20260620 18:38:08.054119 24012 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 25206: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:38:08.054282 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 25206
I20260620 18:38:08.069097 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 25346
Jun 20 18:38:08 dist-test-slave-d645 krb5kdc[25090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980687, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.115.2@KRBTEST.COM
I20260620 18:38:08.083694 25687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:08.069842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60904 (local address 127.23.115.2:35569)
0620 18:38:08.070047 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:08.070053 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:08.070155 (+ 102us) server_negotiation.cc:407] Connection header received
0620 18:38:08.070417 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:08.070423 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:08.070521 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:08.070619 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:08.071796 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:08.072513 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:08.073671 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:08.074121 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:08.074239 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:08.078203 (+ 3964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:08.078231 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:08.078250 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:08.078290 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:08.081113 (+ 2823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:08.081722 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:08.081729 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:08.081731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:08.081851 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:08.082181 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:08.082186 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:08.082190 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:08.082547 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:08.082849 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:08.083124 (+ 275us) server_negotiation.cc:299] Negotiation successful
0620 18:38:08.083329 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260620 18:38:08.084338 25476 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60904
W20260620 18:38:08.084918 24012 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 25346: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:38:08.085067 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 25346
I20260620 18:38:08.107146 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 25485
Jun 20 18:38:08 dist-test-slave-d645 krb5kdc[25090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980687, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.115.3@KRBTEST.COM
I20260620 18:38:08.126231 25638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:08.108353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50366 (local address 127.23.115.3:45969)
0620 18:38:08.108541 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:08.108547 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:08.108566 (+ 19us) server_negotiation.cc:407] Connection header received
0620 18:38:08.108624 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:08.108629 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:08.109679 (+ 1050us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:08.109802 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:08.111420 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:08.112616 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:08.114284 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:08.115077 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:08.115204 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:08.120467 (+ 5263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:08.120481 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:08.120498 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:08.120535 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:08.123491 (+ 2956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:08.124311 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:08.124319 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:08.124321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:08.124453 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:08.124825 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:08.124828 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:08.124830 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:08.125177 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:08.125407 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:08.125711 (+ 304us) server_negotiation.cc:299] Negotiation successful
0620 18:38:08.125934 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260620 18:38:08.127069 25615 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50366
W20260620 18:38:08.127708 24012 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 25485: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:38:08.127868 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 25485
I20260620 18:38:08.145262 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 25103
I20260620 18:38:08.146384 25175 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:38:08.259586 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 25103
2026-06-20T18:38:08Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4319 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980676253068-24012-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:38:08 dist-test-slave-d645 krb5kdc[25710](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:38:08 dist-test-slave-d645 krb5kdc[25710](info): set up 2 sockets
Jun 20 18:38:08 dist-test-slave-d645 krb5kdc[25710](info): commencing operation
krb5kdc: starting...
W20260620 18:38:09.255417 24012 mini_kdc.cc:121] Time spent starting KDC: real 0.957s user 0.000s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980676253068-24012-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980676253068-24012-0/krb5kdc/test-user.keytab.
Jun 20 18:38:09 dist-test-slave-d645 krb5kdc[25710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980689, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:38:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:38:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.115.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.115.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.115.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab.
Entry for principal kudu/127.23.115.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab.
I20260620 18:38:09.427664 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980676253068-24012-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980676253068-24012-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980676253068-24012-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.115.62:45559
--webserver_interface=127.23.115.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab
--principal=kudu/127.23.115.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:41145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.115.62:45559
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:38:09.730343 25723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:09.730655 25723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:09.730790 25723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:09.738112 25723 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:38:09.738234 25723 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 18:38:09.738302 25723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:09.738385 25723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:09.738481 25723 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:38:09.738574 25723 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:38:09.748481 25723 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:41145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980676253068-24012-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980676253068-24012-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.115.62:45559
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab
--principal=kudu/127.23.115.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.115.62:45559
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.115.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980676253068-24012-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:38:09.749737 25723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:09.751009 25723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:09.762096 25728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:09.762056 25731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:09.762209 25729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:09.764096 25723 server_base.cc:1034] running on GCE node
I20260620 18:38:09.764911 25723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:09.766350 25723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:09.767580 25723 hybrid_clock.cc:648] HybridClock initialized: now 1781980689767513 us; error 37 us; skew 500 ppm
I20260620 18:38:09.767928 25723 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:38:09 dist-test-slave-d645 krb5kdc[25710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980689, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:09.771143 25723 init.cc:377] Logged in from keytab as kudu/127.23.115.62@KRBTEST.COM (short username kudu)
I20260620 18:38:09.772699 25723 webserver.cc:460] Webserver started at http://127.23.115.62:33237/ using document root <none> and password file <none>
I20260620 18:38:09.773396 25723 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:09.773526 25723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:09.773842 25723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:38:09.776835 25723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980676253068-24012-0/minicluster-data/master-0/data/instance:
uuid: "7e98328eb2b244639dc33b9f555b6803"
format_stamp: "Formatted at 2026-06-20 18:38:09 on dist-test-slave-d645"
server_key: "cf1e50c8594ba8bf8770d9f9191f8c89"
server_key_iv: "c71ccfe80da7d5e97791c28434472eac"
server_key_version: "encryptionkey@0"
I20260620 18:38:09.777544 25723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980676253068-24012-0/minicluster-data/master-0/wal/instance:
uuid: "7e98328eb2b244639dc33b9f555b6803"
format_stamp: "Formatted at 2026-06-20 18:38:09 on dist-test-slave-d645"
server_key: "cf1e50c8594ba8bf8770d9f9191f8c89"
server_key_iv: "c71ccfe80da7d5e97791c28434472eac"
server_key_version: "encryptionkey@0"
I20260620 18:38:09.782624 25723 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 18:38:09.786222 25738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:09.787571 25723 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:38:09.787729 25723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980676253068-24012-0/minicluster-data/master-0/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980676253068-24012-0/minicluster-data/master-0/wal
uuid: "7e98328eb2b244639dc33b9f555b6803"
format_stamp: "Formatted at 2026-06-20 18:38:09 on dist-test-slave-d645"
server_key: "cf1e50c8594ba8bf8770d9f9191f8c89"
server_key_iv: "c71ccfe80da7d5e97791c28434472eac"
server_key_version: "encryptionkey@0"
I20260620 18:38:09.787918 25723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980676253068-24012-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980676253068-24012-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980676253068-24012-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:09.811054 25723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:09.815215 25723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:09.815465 25723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:09.837519 25723 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.62:45559
I20260620 18:38:09.837553 25799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.62:45559 every 8 connection(s)
I20260620 18:38:09.838860 25723 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
I20260620 18:38:09.842552 25800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:09.843401 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 25723
I20260620 18:38:09.843604 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980676253068-24012-0/minicluster-data/master-0/wal/instance
I20260620 18:38:09.844391 24012 external_mini_cluster.cc:1428] Setting key e5347ae273618295ad5af3d33335a6a3
I20260620 18:38:09.856372 25800 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e98328eb2b244639dc33b9f555b6803: Bootstrap starting.
Jun 20 18:38:09 dist-test-slave-d645 krb5kdc[25710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980689, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.115.62@KRBTEST.COM
I20260620 18:38:09.859913 25800 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e98328eb2b244639dc33b9f555b6803: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:09.860921 25800 log.cc:824] T 00000000000000000000000000000000 P 7e98328eb2b244639dc33b9f555b6803: Log is configured to *not* fsync() on all Append() calls
I20260620 18:38:09.863726 25800 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e98328eb2b244639dc33b9f555b6803: No bootstrap required, opened a new log
I20260620 18:38:09.867064 25803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:09.846449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44986 (local address 127.23.115.62:45559)
0620 18:38:09.847207 (+ 758us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:09.847221 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:09.847264 (+ 43us) server_negotiation.cc:407] Connection header received
0620 18:38:09.848848 (+ 1584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:09.848871 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:09.849248 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:09.849580 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:09.851381 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:09.852730 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:09.854488 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:09.855274 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:09.855423 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:09.859885 (+ 4462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:09.859893 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:09.859910 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:09.859958 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:09.863103 (+ 3145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:09.863746 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:09.863752 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:09.863758 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:09.863941 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:09.864276 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:09.864281 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:09.864285 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:09.864769 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:09.865042 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:09.865460 (+ 418us) server_negotiation.cc:299] Negotiation successful
0620 18:38:09.865866 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":313,"threads_started":1}
I20260620 18:38:09.871505 25800 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e98328eb2b244639dc33b9f555b6803 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e98328eb2b244639dc33b9f555b6803" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 45559 } }
I20260620 18:38:09.871862 25800 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7e98328eb2b244639dc33b9f555b6803 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:09.871973 25800 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e98328eb2b244639dc33b9f555b6803 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e98328eb2b244639dc33b9f555b6803, State: Initialized, Role: FOLLOWER
I20260620 18:38:09.872524 25800 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e98328eb2b244639dc33b9f555b6803 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e98328eb2b244639dc33b9f555b6803" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 45559 } }
I20260620 18:38:09.872732 25800 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7e98328eb2b244639dc33b9f555b6803 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:38:09.872829 25800 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e98328eb2b244639dc33b9f555b6803 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:38:09.872946 25800 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e98328eb2b244639dc33b9f555b6803 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:09.875114 25800 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e98328eb2b244639dc33b9f555b6803 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e98328eb2b244639dc33b9f555b6803" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 45559 } }
I20260620 18:38:09.875587 25800 leader_election.cc:302] T 00000000000000000000000000000000 P 7e98328eb2b244639dc33b9f555b6803 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7e98328eb2b244639dc33b9f555b6803; no voters:
I20260620 18:38:09.876075 25800 leader_election.cc:288] T 00000000000000000000000000000000 P 7e98328eb2b244639dc33b9f555b6803 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:38:09.876217 25805 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e98328eb2b244639dc33b9f555b6803 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:38:09.877336 25805 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7e98328eb2b244639dc33b9f555b6803 [term 1 LEADER]: Becoming Leader. State: Replica: 7e98328eb2b244639dc33b9f555b6803, State: Running, Role: LEADER
I20260620 18:38:09.877839 25805 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e98328eb2b244639dc33b9f555b6803 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e98328eb2b244639dc33b9f555b6803" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 45559 } }
I20260620 18:38:09.878294 25800 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e98328eb2b244639dc33b9f555b6803 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:38:09.880471 25807 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e98328eb2b244639dc33b9f555b6803 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e98328eb2b244639dc33b9f555b6803. Latest consensus state: current_term: 1 leader_uuid: "7e98328eb2b244639dc33b9f555b6803" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e98328eb2b244639dc33b9f555b6803" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 45559 } } }
I20260620 18:38:09.880484 25806 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e98328eb2b244639dc33b9f555b6803 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7e98328eb2b244639dc33b9f555b6803" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e98328eb2b244639dc33b9f555b6803" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 45559 } } }
I20260620 18:38:09.880856 25807 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e98328eb2b244639dc33b9f555b6803 [sys.catalog]: This master's current role is: LEADER
I20260620 18:38:09.880857 25806 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e98328eb2b244639dc33b9f555b6803 [sys.catalog]: This master's current role is: LEADER
I20260620 18:38:09.882879 25813 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:38:09.894042 25813 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:38:09.903101 25813 catalog_manager.cc:1347] Generated new cluster ID: ff31af541954415a9b8cbe492df44efa
I20260620 18:38:09.903211 25813 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:38:09.927299 25813 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:38:09.928669 25813 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:38:09.936781 25813 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7e98328eb2b244639dc33b9f555b6803: Generated new TSK 0
I20260620 18:38:09.937592 25813 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.115.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.115.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.115.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.1.keytab.
Entry for principal kudu/127.23.115.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.1.keytab.
I20260620 18:38:09.995909 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980676253068-24012-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980676253068-24012-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980676253068-24012-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980676253068-24012-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.115.1:0
--local_ip_for_outbound_sockets=127.23.115.1
--webserver_interface=127.23.115.1
--webserver_port=0
--tserver_master_addrs=127.23.115.62:45559
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.1.keytab
--principal=kudu/127.23.115.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:41145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:38:10.344554 25826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:10.344848 25826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:10.344980 25826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:10.352569 25826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:10.352706 25826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:10.352866 25826 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 18:38:10.352933 25826 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 18:38:10.352998 25826 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.115.1
I20260620 18:38:10.363067 25826 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:41145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980676253068-24012-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980676253068-24012-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.1.keytab
--principal=kudu/127.23.115.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.115.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980676253068-24012-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.115.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.115.62:45559
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.115.1
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980676253068-24012-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:38:10.364394 25826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:10.365633 25826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:10.377964 25831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:10.378031 25832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:10.378006 25834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:10.378820 25826 server_base.cc:1034] running on GCE node
I20260620 18:38:10.379431 25826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:10.380292 25826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:10.381491 25826 hybrid_clock.cc:648] HybridClock initialized: now 1781980690381438 us; error 48 us; skew 500 ppm
I20260620 18:38:10.381770 25826 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:38:10 dist-test-slave-d645 krb5kdc[25710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980690, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:10.384847 25826 init.cc:377] Logged in from keytab as kudu/127.23.115.1@KRBTEST.COM (short username kudu)
I20260620 18:38:10.386340 25826 webserver.cc:460] Webserver started at http://127.23.115.1:35281/ using document root <none> and password file <none>
I20260620 18:38:10.387027 25826 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:10.387137 25826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:10.387430 25826 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:38:10.390204 25826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980676253068-24012-0/minicluster-data/ts-0/data/instance:
uuid: "03685ee8dcd84c8db739f4a905afa6d8"
format_stamp: "Formatted at 2026-06-20 18:38:10 on dist-test-slave-d645"
server_key: "efab63a332289a79be9e85e416f54928"
server_key_iv: "ce155e15000c662a0a75384832d24d08"
server_key_version: "encryptionkey@0"
I20260620 18:38:10.390956 25826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980676253068-24012-0/minicluster-data/ts-0/wal/instance:
uuid: "03685ee8dcd84c8db739f4a905afa6d8"
format_stamp: "Formatted at 2026-06-20 18:38:10 on dist-test-slave-d645"
server_key: "efab63a332289a79be9e85e416f54928"
server_key_iv: "ce155e15000c662a0a75384832d24d08"
server_key_version: "encryptionkey@0"
I20260620 18:38:10.395982 25826 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 18:38:10.399297 25841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:10.400552 25826 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 18:38:10.400695 25826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980676253068-24012-0/minicluster-data/ts-0/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980676253068-24012-0/minicluster-data/ts-0/wal
uuid: "03685ee8dcd84c8db739f4a905afa6d8"
format_stamp: "Formatted at 2026-06-20 18:38:10 on dist-test-slave-d645"
server_key: "efab63a332289a79be9e85e416f54928"
server_key_iv: "ce155e15000c662a0a75384832d24d08"
server_key_version: "encryptionkey@0"
I20260620 18:38:10.400868 25826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980676253068-24012-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980676253068-24012-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980676253068-24012-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:10.440253 25826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:10.444679 25826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:10.444984 25826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:10.447690 25826 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:38:10.447809 25826 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:10.447921 25826 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:38:10.448004 25826 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 18:38:10 dist-test-slave-d645 krb5kdc[25710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980690, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.1@KRBTEST.COM for kudu/127.23.115.62@KRBTEST.COM
I20260620 18:38:10.473259 25850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:10.448301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.1:55041 (local address 127.23.115.62:45559)
0620 18:38:10.448898 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:10.448903 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:10.449951 (+ 1048us) server_negotiation.cc:407] Connection header received
0620 18:38:10.451906 (+ 1955us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:10.451912 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:10.451990 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:10.452081 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:10.454104 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:10.455283 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:10.457882 (+ 2599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:10.458524 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:10.458682 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:10.463145 (+ 4463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:10.463150 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:10.463154 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:10.463188 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:10.469170 (+ 5982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:10.470287 (+ 1117us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:10.470291 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:10.470294 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:10.470452 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:10.471107 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:10.471113 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:10.471117 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:10.471447 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:10.471616 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:10.472774 (+ 1158us) server_negotiation.cc:299] Negotiation successful
0620 18:38:10.472971 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":380,"threads_started":1}
I20260620 18:38:10.474756 25851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:10.449012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:45559 (local address 127.23.115.1:55041)
0620 18:38:10.449797 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:10.449835 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:10.451656 (+ 1821us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:10.452329 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:10.452340 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:10.453045 (+ 705us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:10.453835 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:10.453876 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:10.455486 (+ 1610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:10.455492 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:10.457700 (+ 2208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:10.457714 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:10.458824 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:10.458833 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:10.459109 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:10.459862 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:10.459895 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:10.462873 (+ 2978us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:10.469331 (+ 6458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:10.469343 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:10.469364 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:10.470087 (+ 723us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:10.470621 (+ 534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:10.470626 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:10.470630 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:10.470964 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:10.471632 (+ 668us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:10.471641 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:10.472246 (+ 605us) client_negotiation.cc:769] Sending connection context
0620 18:38:10.472625 (+ 379us) client_negotiation.cc:240] Negotiation successful
0620 18:38:10.473229 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":341,"threads_started":1}
I20260620 18:38:10.487847 25826 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.1:41073
I20260620 18:38:10.487886 25961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.1:41073 every 8 connection(s)
I20260620 18:38:10.489451 25826 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980676253068-24012-0/minicluster-data/ts-0/data/info.pb
I20260620 18:38:10.491639 25962 heartbeater.cc:347] Connected to a master server at 127.23.115.62:45559
I20260620 18:38:10.491869 25962 heartbeater.cc:464] Registering TS with master...
I20260620 18:38:10.492668 25962 heartbeater.cc:511] Master 127.23.115.62:45559 requested a full tablet report, sending...
I20260620 18:38:10.494989 25755 ts_manager.cc:194] Registered new tserver with Master: 03685ee8dcd84c8db739f4a905afa6d8 (127.23.115.1:41073)
I20260620 18:38:10.495147 25943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:10.481123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:45559 (local address 127.23.115.1:39319)
0620 18:38:10.482000 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:10.482023 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:10.482197 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:10.482587 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:10.482592 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:10.483239 (+ 647us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:10.483768 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:10.483788 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:10.485129 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:10.485142 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:10.486558 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:10.486573 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:10.487545 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:10.487552 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:10.487751 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:10.488472 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:10.488491 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:10.489251 (+ 760us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:10.492662 (+ 3411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:10.492667 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:10.492671 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:10.493081 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:10.493486 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:10.493492 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:10.493496 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:10.493640 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:10.494295 (+ 655us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:10.494299 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:10.494443 (+ 144us) client_negotiation.cc:769] Sending connection context
0620 18:38:10.494553 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 18:38:10.494776 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":699,"thread_start_us":401,"threads_started":1}
I20260620 18:38:10.495290 25850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:10.481082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.1:39319 (local address 127.23.115.62:45559)
0620 18:38:10.481266 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:10.481273 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:10.482128 (+ 855us) server_negotiation.cc:407] Connection header received
0620 18:38:10.482341 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:10.482348 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:10.482430 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:10.482535 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:10.483981 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:10.484972 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:10.486703 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:10.487341 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:10.487470 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:10.489456 (+ 1986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:10.489461 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:10.489465 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:10.489496 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:10.492516 (+ 3020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:10.493207 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:10.493213 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:10.493217 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:10.493372 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:10.493757 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:10.493761 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:10.493763 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:10.494147 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:10.494292 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:10.494682 (+ 390us) server_negotiation.cc:299] Negotiation successful
0620 18:38:10.494900 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 18:38:10.496465 25755 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.115.1@KRBTEST.COM'} at 127.23.115.1:55041
I20260620 18:38:10.497326 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 25826
I20260620 18:38:10.497524 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980676253068-24012-0/minicluster-data/ts-0/wal/instance
I20260620 18:38:10.498327 24012 external_mini_cluster.cc:1428] Setting key c58149891802b05394b4afce3cdf6302
WARNING: no policy specified for kudu/127.23.115.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.115.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.115.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.2.keytab.
Entry for principal kudu/127.23.115.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.2.keytab.
I20260620 18:38:10.539244 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980676253068-24012-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980676253068-24012-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980676253068-24012-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980676253068-24012-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.115.2:0
--local_ip_for_outbound_sockets=127.23.115.2
--webserver_interface=127.23.115.2
--webserver_port=0
--tserver_master_addrs=127.23.115.62:45559
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.2.keytab
--principal=kudu/127.23.115.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:41145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:38:10.850625 25966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:10.850971 25966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:10.851112 25966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:10.858409 25966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:10.858542 25966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:10.858670 25966 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 18:38:10.858738 25966 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 18:38:10.858856 25966 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.115.2
I20260620 18:38:10.869117 25966 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:41145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980676253068-24012-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980676253068-24012-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.2.keytab
--principal=kudu/127.23.115.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.115.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980676253068-24012-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.115.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.115.62:45559
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.115.2
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980676253068-24012-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:38:10.870406 25966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:10.871909 25966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:10.885612 25971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:10.885614 25974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:10.885685 25972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:10.886253 25966 server_base.cc:1034] running on GCE node
I20260620 18:38:10.886859 25966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:10.887844 25966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:10.889057 25966 hybrid_clock.cc:648] HybridClock initialized: now 1781980690889002 us; error 47 us; skew 500 ppm
I20260620 18:38:10.889387 25966 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:38:10 dist-test-slave-d645 krb5kdc[25710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980690, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:10.892479 25966 init.cc:377] Logged in from keytab as kudu/127.23.115.2@KRBTEST.COM (short username kudu)
I20260620 18:38:10.893863 25966 webserver.cc:460] Webserver started at http://127.23.115.2:41465/ using document root <none> and password file <none>
I20260620 18:38:10.894575 25966 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:10.894692 25966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:10.895022 25966 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:38:10.898141 25966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980676253068-24012-0/minicluster-data/ts-1/data/instance:
uuid: "ef66653e53ec49699727a771ac8d7bb4"
format_stamp: "Formatted at 2026-06-20 18:38:10 on dist-test-slave-d645"
server_key: "9af3f388044c5bda840f86c076d0b793"
server_key_iv: "077c0f9d448b0f1656e3b73756d35bfc"
server_key_version: "encryptionkey@0"
I20260620 18:38:10.899076 25966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980676253068-24012-0/minicluster-data/ts-1/wal/instance:
uuid: "ef66653e53ec49699727a771ac8d7bb4"
format_stamp: "Formatted at 2026-06-20 18:38:10 on dist-test-slave-d645"
server_key: "9af3f388044c5bda840f86c076d0b793"
server_key_iv: "077c0f9d448b0f1656e3b73756d35bfc"
server_key_version: "encryptionkey@0"
I20260620 18:38:10.904635 25966 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 18:38:10.908413 25981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:10.909842 25966 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:38:10.910009 25966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980676253068-24012-0/minicluster-data/ts-1/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980676253068-24012-0/minicluster-data/ts-1/wal
uuid: "ef66653e53ec49699727a771ac8d7bb4"
format_stamp: "Formatted at 2026-06-20 18:38:10 on dist-test-slave-d645"
server_key: "9af3f388044c5bda840f86c076d0b793"
server_key_iv: "077c0f9d448b0f1656e3b73756d35bfc"
server_key_version: "encryptionkey@0"
I20260620 18:38:10.910208 25966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980676253068-24012-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980676253068-24012-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980676253068-24012-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:10.932211 25966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:10.936808 25966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:10.937083 25966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:10.939733 25966 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:38:10.939841 25966 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:10.939949 25966 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:38:10.940026 25966 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 18:38:10 dist-test-slave-d645 krb5kdc[25710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980690, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.2@KRBTEST.COM for kudu/127.23.115.62@KRBTEST.COM
I20260620 18:38:10.963816 25850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:10.940254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.2:57467 (local address 127.23.115.62:45559)
0620 18:38:10.940437 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:10.940443 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:10.941861 (+ 1418us) server_negotiation.cc:407] Connection header received
0620 18:38:10.944080 (+ 2219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:10.944086 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:10.944179 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:10.944254 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:10.946620 (+ 2366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:10.947605 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:10.950455 (+ 2850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:10.951026 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:10.951152 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:10.956255 (+ 5103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:10.956262 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:10.956267 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:10.956309 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:10.959698 (+ 3389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:10.960725 (+ 1027us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:10.960730 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:10.960734 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:10.960887 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:10.961512 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:10.961517 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:10.961520 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:10.961846 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:10.962605 (+ 759us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:10.963178 (+ 573us) server_negotiation.cc:299] Negotiation successful
0620 18:38:10.963428 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 18:38:10.965560 25990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:10.940817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:45559 (local address 127.23.115.2:57467)
0620 18:38:10.941650 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:10.941719 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:10.943753 (+ 2034us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:10.944815 (+ 1062us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:10.944831 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:10.945571 (+ 740us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:10.946439 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:10.946460 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:10.947793 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:10.947800 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:10.950273 (+ 2473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:10.950297 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:10.951308 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:10.951317 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:10.951628 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:10.952523 (+ 895us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:10.952567 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:10.955985 (+ 3418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:10.959891 (+ 3906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:10.959901 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:10.959926 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:10.960600 (+ 674us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:10.961031 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:10.961036 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:10.961045 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:10.961371 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:10.962013 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:10.962023 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:10.962659 (+ 636us) client_negotiation.cc:769] Sending connection context
0620 18:38:10.963074 (+ 415us) client_negotiation.cc:240] Negotiation successful
0620 18:38:10.963748 (+ 674us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":351,"threads_started":1}
I20260620 18:38:10.978142 25966 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.2:36169
I20260620 18:38:10.978255 26100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.2:36169 every 8 connection(s)
I20260620 18:38:10.979652 25966 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980676253068-24012-0/minicluster-data/ts-1/data/info.pb
I20260620 18:38:10.981942 26101 heartbeater.cc:347] Connected to a master server at 127.23.115.62:45559
I20260620 18:38:10.982192 26101 heartbeater.cc:464] Registering TS with master...
I20260620 18:38:10.983158 26101 heartbeater.cc:511] Master 127.23.115.62:45559 requested a full tablet report, sending...
I20260620 18:38:10.984570 25755 ts_manager.cc:194] Registered new tserver with Master: ef66653e53ec49699727a771ac8d7bb4 (127.23.115.2:36169)
I20260620 18:38:10.985350 25755 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.115.2@KRBTEST.COM'} at 127.23.115.2:57467
I20260620 18:38:10.986186 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 25966
I20260620 18:38:10.986361 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980676253068-24012-0/minicluster-data/ts-1/wal/instance
I20260620 18:38:10.986899 24012 external_mini_cluster.cc:1428] Setting key b0d9d9a22e6671f0ae25acea5cfa9db9
I20260620 18:38:10.991616 26082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:10.971448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:45559 (local address 127.23.115.2:36249)
0620 18:38:10.978545 (+ 7097us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:10.978567 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:10.978735 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:10.979073 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:10.979078 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:10.979568 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:10.979944 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:10.979957 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:10.981228 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:10.981234 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:10.982890 (+ 1656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:10.982906 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:10.983911 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:10.983918 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:10.984114 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:10.984678 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:10.984734 (+ 56us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:10.985403 (+ 669us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:10.989026 (+ 3623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:10.989030 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:10.989033 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:10.989419 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:10.989792 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:10.989795 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:10.989797 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:10.989906 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:10.990643 (+ 737us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:10.990646 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:10.990831 (+ 185us) client_negotiation.cc:769] Sending connection context
0620 18:38:10.991003 (+ 172us) client_negotiation.cc:240] Negotiation successful
0620 18:38:10.991235 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6934,"thread_start_us":6576,"threads_started":1}
I20260620 18:38:10.991676 25850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:10.971318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.2:36249 (local address 127.23.115.62:45559)
0620 18:38:10.971499 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:10.971505 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:10.978666 (+ 7161us) server_negotiation.cc:407] Connection header received
0620 18:38:10.978867 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:10.978871 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:10.978962 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:10.979048 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:10.980086 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:10.981103 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:10.983105 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:10.983713 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:10.983856 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:10.985600 (+ 1744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:10.985607 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:10.985611 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:10.985644 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:10.988886 (+ 3242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:10.989537 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:10.989543 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:10.989546 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:10.989691 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:10.989989 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:10.989994 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:10.989997 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:10.990512 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:10.990695 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:10.991126 (+ 431us) server_negotiation.cc:299] Negotiation successful
0620 18:38:10.991367 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
WARNING: no policy specified for kudu/127.23.115.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.115.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.115.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.3.keytab.
Entry for principal kudu/127.23.115.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.3.keytab.
I20260620 18:38:11.027218 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980676253068-24012-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980676253068-24012-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980676253068-24012-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980676253068-24012-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.115.3:0
--local_ip_for_outbound_sockets=127.23.115.3
--webserver_interface=127.23.115.3
--webserver_port=0
--tserver_master_addrs=127.23.115.62:45559
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.3.keytab
--principal=kudu/127.23.115.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:41145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:38:11.335700 26105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:11.335961 26105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:11.336058 26105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:11.343111 26105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:11.343214 26105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:11.343293 26105 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 18:38:11.343345 26105 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 18:38:11.343405 26105 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.115.3
I20260620 18:38:11.353013 26105 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:41145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980676253068-24012-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980676253068-24012-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.3.keytab
--principal=kudu/127.23.115.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.115.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980676253068-24012-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.115.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.115.62:45559
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.115.3
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980676253068-24012-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:38:11.354229 26105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:11.355468 26105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:11.368006 26111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:11.368095 26110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:11.368178 26113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:11.369253 26105 server_base.cc:1034] running on GCE node
I20260620 18:38:11.369827 26105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:11.370695 26105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:11.371884 26105 hybrid_clock.cc:648] HybridClock initialized: now 1781980691371809 us; error 59 us; skew 500 ppm
I20260620 18:38:11.372189 26105 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:38:11 dist-test-slave-d645 krb5kdc[25710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980691, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:11.375234 26105 init.cc:377] Logged in from keytab as kudu/127.23.115.3@KRBTEST.COM (short username kudu)
I20260620 18:38:11.376672 26105 webserver.cc:460] Webserver started at http://127.23.115.3:36929/ using document root <none> and password file <none>
I20260620 18:38:11.377349 26105 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:11.377466 26105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:11.377754 26105 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:38:11.380692 26105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980676253068-24012-0/minicluster-data/ts-2/data/instance:
uuid: "e9e5bea8c365407db364bf7b7aebf653"
format_stamp: "Formatted at 2026-06-20 18:38:11 on dist-test-slave-d645"
server_key: "938359a55b938354333c82dbbe95eae9"
server_key_iv: "d652ded2bb1cf2648468dc3bfcde2687"
server_key_version: "encryptionkey@0"
I20260620 18:38:11.381417 26105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980676253068-24012-0/minicluster-data/ts-2/wal/instance:
uuid: "e9e5bea8c365407db364bf7b7aebf653"
format_stamp: "Formatted at 2026-06-20 18:38:11 on dist-test-slave-d645"
server_key: "938359a55b938354333c82dbbe95eae9"
server_key_iv: "d652ded2bb1cf2648468dc3bfcde2687"
server_key_version: "encryptionkey@0"
I20260620 18:38:11.386433 26105 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260620 18:38:11.389896 26120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:11.391482 26105 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:38:11.391680 26105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980676253068-24012-0/minicluster-data/ts-2/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980676253068-24012-0/minicluster-data/ts-2/wal
uuid: "e9e5bea8c365407db364bf7b7aebf653"
format_stamp: "Formatted at 2026-06-20 18:38:11 on dist-test-slave-d645"
server_key: "938359a55b938354333c82dbbe95eae9"
server_key_iv: "d652ded2bb1cf2648468dc3bfcde2687"
server_key_version: "encryptionkey@0"
I20260620 18:38:11.391865 26105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980676253068-24012-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980676253068-24012-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980676253068-24012-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:11.404125 26105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:11.408599 26105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:11.408876 26105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:11.411548 26105 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:38:11.411664 26105 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:11.411811 26105 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:38:11.411893 26105 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 18:38:11 dist-test-slave-d645 krb5kdc[25710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980691, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.3@KRBTEST.COM for kudu/127.23.115.62@KRBTEST.COM
I20260620 18:38:11.434325 25850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:11.411952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.3:51063 (local address 127.23.115.62:45559)
0620 18:38:11.412116 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:11.412123 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:11.413679 (+ 1556us) server_negotiation.cc:407] Connection header received
0620 18:38:11.415891 (+ 2212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:11.415897 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:11.415990 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:11.416075 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:11.418102 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:11.419263 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:11.422048 (+ 2785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:11.422577 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:11.422739 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:11.427198 (+ 4459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:11.427205 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:11.427208 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:11.427243 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:11.430635 (+ 3392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:11.431607 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:11.431613 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:11.431617 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:11.431786 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:11.432401 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:11.432407 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:11.432410 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:11.432732 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:11.432883 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:11.433746 (+ 863us) server_negotiation.cc:299] Negotiation successful
0620 18:38:11.433973 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 18:38:11.435484 26129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:11.412642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:45559 (local address 127.23.115.3:51063)
0620 18:38:11.413526 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:11.413565 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:11.415622 (+ 2057us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:11.416358 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:11.416370 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:11.417115 (+ 745us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:11.417903 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:11.417929 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:11.419505 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:11.419513 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:11.421879 (+ 2366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:11.421896 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:11.423022 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:11.423028 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:11.423242 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:11.423983 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:11.424017 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:11.426919 (+ 2902us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:11.430817 (+ 3898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:11.430826 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:11.430843 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:11.431490 (+ 647us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:11.431920 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:11.431929 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:11.431934 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:11.432257 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:11.432970 (+ 713us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:11.432979 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:11.433358 (+ 379us) client_negotiation.cc:769] Sending connection context
0620 18:38:11.433596 (+ 238us) client_negotiation.cc:240] Negotiation successful
0620 18:38:11.434106 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":386,"threads_started":1}
I20260620 18:38:11.454663 25850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:11.441366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.3:56743 (local address 127.23.115.62:45559)
0620 18:38:11.441773 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:11.441778 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:11.442191 (+ 413us) server_negotiation.cc:407] Connection header received
0620 18:38:11.442381 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:11.442384 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:11.442444 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:11.442514 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:11.443688 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:11.444367 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:11.446225 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:11.446616 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:11.446702 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:11.448913 (+ 2211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:11.448918 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:11.448921 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:11.448950 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:11.451966 (+ 3016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:11.452620 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:11.452626 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:11.452630 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:11.452777 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:11.453114 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:11.453119 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:11.453123 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:11.453475 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:11.453626 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:11.454078 (+ 452us) server_negotiation.cc:299] Negotiation successful
0620 18:38:11.454293 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308}
I20260620 18:38:11.454712 26209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:11.441510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:45559 (local address 127.23.115.3:56743)
0620 18:38:11.442085 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:11.442107 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:11.442260 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:11.442554 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:11.442559 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:11.443168 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:11.443555 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:11.443567 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:11.444475 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:11.444480 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:11.446093 (+ 1613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:11.446107 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:11.446776 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:11.446783 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:11.447002 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:11.447676 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:11.447699 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:11.448723 (+ 1024us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:11.452129 (+ 3406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:11.452134 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:11.452138 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:11.452516 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:11.452894 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:11.452899 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:11.452901 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:11.453016 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:11.453613 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:11.453618 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:11.453821 (+ 203us) client_negotiation.cc:769] Sending connection context
0620 18:38:11.453954 (+ 133us) client_negotiation.cc:240] Negotiation successful
0620 18:38:11.454306 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":283,"threads_started":1}
I20260620 18:38:11.461635 26105 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.3:38119
I20260620 18:38:11.461670 26239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.3:38119 every 8 connection(s)
I20260620 18:38:11.462797 26105 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980676253068-24012-0/minicluster-data/ts-2/data/info.pb
I20260620 18:38:11.463649 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 26105
I20260620 18:38:11.463820 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980676253068-24012-0/minicluster-data/ts-2/wal/instance
I20260620 18:38:11.464315 24012 external_mini_cluster.cc:1428] Setting key b9a9738f71b9a97e1916a8f194bfc0c3
I20260620 18:38:11.464936 26240 heartbeater.cc:347] Connected to a master server at 127.23.115.62:45559
I20260620 18:38:11.465179 26240 heartbeater.cc:464] Registering TS with master...
I20260620 18:38:11.465936 26240 heartbeater.cc:511] Master 127.23.115.62:45559 requested a full tablet report, sending...
I20260620 18:38:11.467393 25755 ts_manager.cc:194] Registered new tserver with Master: e9e5bea8c365407db364bf7b7aebf653 (127.23.115.3:38119)
I20260620 18:38:11.468209 25755 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.115.3@KRBTEST.COM'} at 127.23.115.3:51063
I20260620 18:38:11.479844 24012 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:38:11.499858 25962 heartbeater.cc:503] Master 127.23.115.62:45559 was elected leader, sending a full tablet report...
Jun 20 18:38:11 dist-test-slave-d645 krb5kdc[25710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980691, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 18:38:11 dist-test-slave-d645 krb5kdc[25710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980691, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.115.62@KRBTEST.COM
I20260620 18:38:11.523608 25850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:11.510648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45000 (local address 127.23.115.62:45559)
0620 18:38:11.510873 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:11.510878 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:11.511031 (+ 153us) server_negotiation.cc:407] Connection header received
0620 18:38:11.511215 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:11.511220 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:11.511283 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:11.511362 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:11.512684 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:11.513425 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:11.514668 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:11.515154 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:11.515261 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:11.518724 (+ 3463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:11.518729 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:11.518732 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:11.518764 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:11.521310 (+ 2546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:11.521863 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:11.521867 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:11.521869 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:11.521995 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:11.522350 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:11.522355 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:11.522357 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:11.522637 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:11.522823 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:11.523127 (+ 304us) server_negotiation.cc:299] Negotiation successful
0620 18:38:11.523311 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260620 18:38:11.527544 25755 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45000:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:38:11.534138 25755 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:38:11.554832 26255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:11.545707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46998 (local address 127.23.115.3:38119)
0620 18:38:11.547552 (+ 1845us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:11.547560 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:11.547593 (+ 33us) server_negotiation.cc:407] Connection header received
0620 18:38:11.547683 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:11.547691 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:11.547923 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:11.548525 (+ 602us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:38:11.548860 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:11.550304 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:11.552436 (+ 2132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:11.553830 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:11.553938 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:11.554164 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:11.554384 (+ 220us) server_negotiation.cc:299] Negotiation successful
0620 18:38:11.554533 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1689,"thread_start_us":1530,"threads_started":1}
I20260620 18:38:11.554888 26253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:11.545584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.3:38119 (local address 127.0.0.1:46998)
0620 18:38:11.546227 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:11.546286 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:11.546441 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:11.548155 (+ 1714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:11.548162 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:11.548206 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:38:11.548656 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:11.548668 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:11.550446 (+ 1778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:11.550450 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:11.552279 (+ 1829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:11.552297 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:11.553996 (+ 1699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:11.554001 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:11.554178 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:11.554225 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 18:38:11.554331 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 18:38:11.554510 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"mutex_wait_us":15,"thread_start_us":1339,"threads_started":1}
I20260620 18:38:11.556018 26257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:11.546955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.2:36169 (local address 127.0.0.1:48288)
0620 18:38:11.548036 (+ 1081us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:11.548057 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:11.548163 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:11.549005 (+ 842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:11.549009 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:11.549030 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:38:11.549573 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:11.549582 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:11.551465 (+ 1883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:11.551470 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:11.553586 (+ 2116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:11.553597 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:11.555231 (+ 1634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:11.555236 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:11.555397 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:11.555430 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 18:38:11.555496 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 18:38:11.555621 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":975,"mutex_wait_us":57,"thread_start_us":859,"threads_started":1}
I20260620 18:38:11.556526 26254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:11.545612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.1:41073 (local address 127.0.0.1:50466)
0620 18:38:11.546975 (+ 1363us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:11.546999 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:11.547104 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:11.548280 (+ 1176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:11.548284 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:11.548304 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:38:11.549241 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:11.549259 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:11.551526 (+ 2267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:11.551530 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:11.553627 (+ 2097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:11.553641 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:11.555663 (+ 2022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:11.555669 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:11.555837 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:11.555872 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 18:38:11.555962 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 18:38:11.556086 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1204,"mutex_wait_us":101,"thread_start_us":587,"threads_started":1}
I20260620 18:38:11.556888 26258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:11.547419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48288 (local address 127.23.115.2:36169)
0620 18:38:11.548609 (+ 1190us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:11.548614 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:11.548634 (+ 20us) server_negotiation.cc:407] Connection header received
0620 18:38:11.548697 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:11.548702 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:11.548848 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:11.548990 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:38:11.549853 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:11.551327 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:11.553729 (+ 2402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:11.555014 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:11.555885 (+ 871us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:11.556068 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:11.556315 (+ 247us) server_negotiation.cc:299] Negotiation successful
0620 18:38:11.556461 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1068,"thread_start_us":955,"threads_started":1}
I20260620 18:38:11.556972 26256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:11.546076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50466 (local address 127.23.115.1:41073)
0620 18:38:11.547790 (+ 1714us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:11.547798 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:11.547820 (+ 22us) server_negotiation.cc:407] Connection header received
0620 18:38:11.547900 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:11.547906 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:11.548097 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:11.549111 (+ 1014us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:38:11.549888 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:11.551327 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:11.553765 (+ 2438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:11.555467 (+ 1702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:11.555591 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:11.555776 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:11.556242 (+ 466us) server_negotiation.cc:299] Negotiation successful
0620 18:38:11.556399 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1564,"thread_start_us":1443,"threads_started":1}
I20260620 18:38:11.558607 26170 tablet_service.cc:1467] Processing CreateTablet for tablet 8d42299735354b44b03cc58f6b02d6aa (DEFAULT_TABLE table=test-table [id=c74afcfde1844d398f501c92c1b837ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:38:11.559561 26031 tablet_service.cc:1467] Processing CreateTablet for tablet 8d42299735354b44b03cc58f6b02d6aa (DEFAULT_TABLE table=test-table [id=c74afcfde1844d398f501c92c1b837ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:38:11.560117 26170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d42299735354b44b03cc58f6b02d6aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:11.560047 25892 tablet_service.cc:1467] Processing CreateTablet for tablet 8d42299735354b44b03cc58f6b02d6aa (DEFAULT_TABLE table=test-table [id=c74afcfde1844d398f501c92c1b837ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:38:11.560606 26031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d42299735354b44b03cc58f6b02d6aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:11.561210 25892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d42299735354b44b03cc58f6b02d6aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:11.571412 26259 tablet_bootstrap.cc:492] T 8d42299735354b44b03cc58f6b02d6aa P ef66653e53ec49699727a771ac8d7bb4: Bootstrap starting.
I20260620 18:38:11.574370 26259 tablet_bootstrap.cc:654] T 8d42299735354b44b03cc58f6b02d6aa P ef66653e53ec49699727a771ac8d7bb4: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:11.575441 26260 tablet_bootstrap.cc:492] T 8d42299735354b44b03cc58f6b02d6aa P 03685ee8dcd84c8db739f4a905afa6d8: Bootstrap starting.
I20260620 18:38:11.576447 26261 tablet_bootstrap.cc:492] T 8d42299735354b44b03cc58f6b02d6aa P e9e5bea8c365407db364bf7b7aebf653: Bootstrap starting.
I20260620 18:38:11.577286 26259 log.cc:824] T 8d42299735354b44b03cc58f6b02d6aa P ef66653e53ec49699727a771ac8d7bb4: Log is configured to *not* fsync() on all Append() calls
I20260620 18:38:11.579531 26260 tablet_bootstrap.cc:654] T 8d42299735354b44b03cc58f6b02d6aa P 03685ee8dcd84c8db739f4a905afa6d8: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:11.580448 26261 tablet_bootstrap.cc:654] T 8d42299735354b44b03cc58f6b02d6aa P e9e5bea8c365407db364bf7b7aebf653: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:11.580765 26260 log.cc:824] T 8d42299735354b44b03cc58f6b02d6aa P 03685ee8dcd84c8db739f4a905afa6d8: Log is configured to *not* fsync() on all Append() calls
I20260620 18:38:11.580914 26259 tablet_bootstrap.cc:492] T 8d42299735354b44b03cc58f6b02d6aa P ef66653e53ec49699727a771ac8d7bb4: No bootstrap required, opened a new log
I20260620 18:38:11.581185 26259 ts_tablet_manager.cc:1397] T 8d42299735354b44b03cc58f6b02d6aa P ef66653e53ec49699727a771ac8d7bb4: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.005s
I20260620 18:38:11.581486 26261 log.cc:824] T 8d42299735354b44b03cc58f6b02d6aa P e9e5bea8c365407db364bf7b7aebf653: Log is configured to *not* fsync() on all Append() calls
I20260620 18:38:11.583913 26260 tablet_bootstrap.cc:492] T 8d42299735354b44b03cc58f6b02d6aa P 03685ee8dcd84c8db739f4a905afa6d8: No bootstrap required, opened a new log
I20260620 18:38:11.584170 26260 ts_tablet_manager.cc:1397] T 8d42299735354b44b03cc58f6b02d6aa P 03685ee8dcd84c8db739f4a905afa6d8: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260620 18:38:11.589021 26261 tablet_bootstrap.cc:492] T 8d42299735354b44b03cc58f6b02d6aa P e9e5bea8c365407db364bf7b7aebf653: No bootstrap required, opened a new log
I20260620 18:38:11.589274 26261 ts_tablet_manager.cc:1397] T 8d42299735354b44b03cc58f6b02d6aa P e9e5bea8c365407db364bf7b7aebf653: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.009s
I20260620 18:38:11.591333 26259 raft_consensus.cc:348] T 8d42299735354b44b03cc58f6b02d6aa P ef66653e53ec49699727a771ac8d7bb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9e5bea8c365407db364bf7b7aebf653" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 38119 } } peers { permanent_uuid: "03685ee8dcd84c8db739f4a905afa6d8" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 41073 } } peers { permanent_uuid: "ef66653e53ec49699727a771ac8d7bb4" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 36169 } }
I20260620 18:38:11.591773 26259 raft_consensus.cc:374] T 8d42299735354b44b03cc58f6b02d6aa P ef66653e53ec49699727a771ac8d7bb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:11.591868 26259 raft_consensus.cc:729] T 8d42299735354b44b03cc58f6b02d6aa P ef66653e53ec49699727a771ac8d7bb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef66653e53ec49699727a771ac8d7bb4, State: Initialized, Role: FOLLOWER
I20260620 18:38:11.592468 26259 consensus_queue.cc:260] T 8d42299735354b44b03cc58f6b02d6aa P ef66653e53ec49699727a771ac8d7bb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9e5bea8c365407db364bf7b7aebf653" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 38119 } } peers { permanent_uuid: "03685ee8dcd84c8db739f4a905afa6d8" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 41073 } } peers { permanent_uuid: "ef66653e53ec49699727a771ac8d7bb4" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 36169 } }
I20260620 18:38:11.593554 26101 heartbeater.cc:503] Master 127.23.115.62:45559 was elected leader, sending a full tablet report...
I20260620 18:38:11.594190 26259 ts_tablet_manager.cc:1428] T 8d42299735354b44b03cc58f6b02d6aa P ef66653e53ec49699727a771ac8d7bb4: Time spent starting tablet: real 0.013s user 0.004s sys 0.010s
I20260620 18:38:11.594275 26260 raft_consensus.cc:348] T 8d42299735354b44b03cc58f6b02d6aa P 03685ee8dcd84c8db739f4a905afa6d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9e5bea8c365407db364bf7b7aebf653" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 38119 } } peers { permanent_uuid: "03685ee8dcd84c8db739f4a905afa6d8" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 41073 } } peers { permanent_uuid: "ef66653e53ec49699727a771ac8d7bb4" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 36169 } }
I20260620 18:38:11.594718 26260 raft_consensus.cc:374] T 8d42299735354b44b03cc58f6b02d6aa P 03685ee8dcd84c8db739f4a905afa6d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:11.594818 26260 raft_consensus.cc:729] T 8d42299735354b44b03cc58f6b02d6aa P 03685ee8dcd84c8db739f4a905afa6d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03685ee8dcd84c8db739f4a905afa6d8, State: Initialized, Role: FOLLOWER
I20260620 18:38:11.595571 26260 consensus_queue.cc:260] T 8d42299735354b44b03cc58f6b02d6aa P 03685ee8dcd84c8db739f4a905afa6d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9e5bea8c365407db364bf7b7aebf653" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 38119 } } peers { permanent_uuid: "03685ee8dcd84c8db739f4a905afa6d8" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 41073 } } peers { permanent_uuid: "ef66653e53ec49699727a771ac8d7bb4" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 36169 } }
I20260620 18:38:11.597325 26260 ts_tablet_manager.cc:1428] T 8d42299735354b44b03cc58f6b02d6aa P 03685ee8dcd84c8db739f4a905afa6d8: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
I20260620 18:38:11.597927 26261 raft_consensus.cc:348] T 8d42299735354b44b03cc58f6b02d6aa P e9e5bea8c365407db364bf7b7aebf653 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9e5bea8c365407db364bf7b7aebf653" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 38119 } } peers { permanent_uuid: "03685ee8dcd84c8db739f4a905afa6d8" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 41073 } } peers { permanent_uuid: "ef66653e53ec49699727a771ac8d7bb4" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 36169 } }
I20260620 18:38:11.598313 26261 raft_consensus.cc:374] T 8d42299735354b44b03cc58f6b02d6aa P e9e5bea8c365407db364bf7b7aebf653 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:11.598407 26261 raft_consensus.cc:729] T 8d42299735354b44b03cc58f6b02d6aa P e9e5bea8c365407db364bf7b7aebf653 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9e5bea8c365407db364bf7b7aebf653, State: Initialized, Role: FOLLOWER
I20260620 18:38:11.598986 26261 consensus_queue.cc:260] T 8d42299735354b44b03cc58f6b02d6aa P e9e5bea8c365407db364bf7b7aebf653 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9e5bea8c365407db364bf7b7aebf653" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 38119 } } peers { permanent_uuid: "03685ee8dcd84c8db739f4a905afa6d8" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 41073 } } peers { permanent_uuid: "ef66653e53ec49699727a771ac8d7bb4" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 36169 } }
I20260620 18:38:11.600216 26240 heartbeater.cc:503] Master 127.23.115.62:45559 was elected leader, sending a full tablet report...
I20260620 18:38:11.600622 26261 ts_tablet_manager.cc:1428] T 8d42299735354b44b03cc58f6b02d6aa P e9e5bea8c365407db364bf7b7aebf653: Time spent starting tablet: real 0.011s user 0.006s sys 0.006s
W20260620 18:38:11.714346 26241 tablet.cc:2369] T 8d42299735354b44b03cc58f6b02d6aa P e9e5bea8c365407db364bf7b7aebf653: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:38:11.731845 26102 tablet.cc:2369] T 8d42299735354b44b03cc58f6b02d6aa P ef66653e53ec49699727a771ac8d7bb4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:38:11.742085 25963 tablet.cc:2369] T 8d42299735354b44b03cc58f6b02d6aa P 03685ee8dcd84c8db739f4a905afa6d8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:38:11.743223 26265 raft_consensus.cc:482] T 8d42299735354b44b03cc58f6b02d6aa P ef66653e53ec49699727a771ac8d7bb4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:38:11.743521 26265 raft_consensus.cc:504] T 8d42299735354b44b03cc58f6b02d6aa P ef66653e53ec49699727a771ac8d7bb4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9e5bea8c365407db364bf7b7aebf653" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 38119 } } peers { permanent_uuid: "03685ee8dcd84c8db739f4a905afa6d8" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 41073 } } peers { permanent_uuid: "ef66653e53ec49699727a771ac8d7bb4" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 36169 } }
I20260620 18:38:11.745096 26265 leader_election.cc:288] T 8d42299735354b44b03cc58f6b02d6aa P ef66653e53ec49699727a771ac8d7bb4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e9e5bea8c365407db364bf7b7aebf653 (127.23.115.3:38119), 03685ee8dcd84c8db739f4a905afa6d8 (127.23.115.1:41073)
I20260620 18:38:11.751257 26256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:11.745582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.2:35889 (local address 127.23.115.1:41073)
0620 18:38:11.745817 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:11.745823 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:11.746173 (+ 350us) server_negotiation.cc:407] Connection header received
0620 18:38:11.746365 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:11.746369 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:11.746437 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:11.746511 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:38:11.747012 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:11.747857 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:11.749604 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:11.750357 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:11.750514 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:11.750594 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:11.750865 (+ 271us) server_negotiation.cc:299] Negotiation successful
0620 18:38:11.751019 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260620 18:38:11.751358 26268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:11.745475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.1:41073 (local address 127.23.115.2:35889)
0620 18:38:11.746065 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:11.746087 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:11.746237 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:11.746545 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:11.746550 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:11.746583 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:38:11.746872 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:11.746880 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:11.747995 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:11.747999 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:11.749449 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:11.749462 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:11.750561 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:11.750576 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:11.750718 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:11.750742 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 18:38:11.750823 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 18:38:11.750987 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":350,"threads_started":1}
I20260620 18:38:11.751586 26255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:11.745499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.2:50405 (local address 127.23.115.3:38119)
0620 18:38:11.745708 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:11.745715 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:11.746599 (+ 884us) server_negotiation.cc:407] Connection header received
0620 18:38:11.746779 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:11.746784 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:11.746843 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:11.746920 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:38:11.747447 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:11.748239 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:11.749908 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:11.750644 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:11.750803 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:11.750880 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:11.751225 (+ 345us) server_negotiation.cc:299] Negotiation successful
0620 18:38:11.751352 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 18:38:11.751840 26269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:11.745526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.3:38119 (local address 127.23.115.2:50405)
0620 18:38:11.746499 (+ 973us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:11.746513 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:11.746642 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:11.746972 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:11.746976 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:11.746998 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:38:11.747306 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:11.747315 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:11.748393 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:11.748397 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:11.749756 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:11.749768 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:11.750919 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:11.750925 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:11.751082 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:11.751104 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 18:38:11.751198 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 18:38:11.751318 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":863,"thread_start_us":727,"threads_started":1}
I20260620 18:38:11.752337 25912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d42299735354b44b03cc58f6b02d6aa" candidate_uuid: "ef66653e53ec49699727a771ac8d7bb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03685ee8dcd84c8db739f4a905afa6d8" is_pre_election: true
I20260620 18:38:11.752612 26190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d42299735354b44b03cc58f6b02d6aa" candidate_uuid: "ef66653e53ec49699727a771ac8d7bb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9e5bea8c365407db364bf7b7aebf653" is_pre_election: true
I20260620 18:38:11.752807 25912 raft_consensus.cc:2454] T 8d42299735354b44b03cc58f6b02d6aa P 03685ee8dcd84c8db739f4a905afa6d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef66653e53ec49699727a771ac8d7bb4 in term 0.
I20260620 18:38:11.753062 26190 raft_consensus.cc:2454] T 8d42299735354b44b03cc58f6b02d6aa P e9e5bea8c365407db364bf7b7aebf653 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef66653e53ec49699727a771ac8d7bb4 in term 0.
I20260620 18:38:11.753458 25982 leader_election.cc:302] T 8d42299735354b44b03cc58f6b02d6aa P ef66653e53ec49699727a771ac8d7bb4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03685ee8dcd84c8db739f4a905afa6d8, ef66653e53ec49699727a771ac8d7bb4; no voters:
I20260620 18:38:11.753873 26265 raft_consensus.cc:2781] T 8d42299735354b44b03cc58f6b02d6aa P ef66653e53ec49699727a771ac8d7bb4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:38:11.754021 26265 raft_consensus.cc:482] T 8d42299735354b44b03cc58f6b02d6aa P ef66653e53ec49699727a771ac8d7bb4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:38:11.754112 26265 raft_consensus.cc:3037] T 8d42299735354b44b03cc58f6b02d6aa P ef66653e53ec49699727a771ac8d7bb4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:11.756273 26265 raft_consensus.cc:504] T 8d42299735354b44b03cc58f6b02d6aa P ef66653e53ec49699727a771ac8d7bb4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9e5bea8c365407db364bf7b7aebf653" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 38119 } } peers { permanent_uuid: "03685ee8dcd84c8db739f4a905afa6d8" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 41073 } } peers { permanent_uuid: "ef66653e53ec49699727a771ac8d7bb4" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 36169 } }
I20260620 18:38:11.756841 26265 leader_election.cc:288] T 8d42299735354b44b03cc58f6b02d6aa P ef66653e53ec49699727a771ac8d7bb4 [CANDIDATE]: Term 1 election: Requested vote from peers e9e5bea8c365407db364bf7b7aebf653 (127.23.115.3:38119), 03685ee8dcd84c8db739f4a905afa6d8 (127.23.115.1:41073)
I20260620 18:38:11.757418 25912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d42299735354b44b03cc58f6b02d6aa" candidate_uuid: "ef66653e53ec49699727a771ac8d7bb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03685ee8dcd84c8db739f4a905afa6d8"
I20260620 18:38:11.757408 26190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d42299735354b44b03cc58f6b02d6aa" candidate_uuid: "ef66653e53ec49699727a771ac8d7bb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9e5bea8c365407db364bf7b7aebf653"
I20260620 18:38:11.757645 25912 raft_consensus.cc:3037] T 8d42299735354b44b03cc58f6b02d6aa P 03685ee8dcd84c8db739f4a905afa6d8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:11.757643 26190 raft_consensus.cc:3037] T 8d42299735354b44b03cc58f6b02d6aa P e9e5bea8c365407db364bf7b7aebf653 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:11.760730 25912 raft_consensus.cc:2454] T 8d42299735354b44b03cc58f6b02d6aa P 03685ee8dcd84c8db739f4a905afa6d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef66653e53ec49699727a771ac8d7bb4 in term 1.
I20260620 18:38:11.760730 26190 raft_consensus.cc:2454] T 8d42299735354b44b03cc58f6b02d6aa P e9e5bea8c365407db364bf7b7aebf653 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef66653e53ec49699727a771ac8d7bb4 in term 1.
I20260620 18:38:11.761173 25982 leader_election.cc:302] T 8d42299735354b44b03cc58f6b02d6aa P ef66653e53ec49699727a771ac8d7bb4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 03685ee8dcd84c8db739f4a905afa6d8, e9e5bea8c365407db364bf7b7aebf653, ef66653e53ec49699727a771ac8d7bb4; no voters:
I20260620 18:38:11.761505 26265 raft_consensus.cc:2781] T 8d42299735354b44b03cc58f6b02d6aa P ef66653e53ec49699727a771ac8d7bb4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:38:11.761962 26265 raft_consensus.cc:686] T 8d42299735354b44b03cc58f6b02d6aa P ef66653e53ec49699727a771ac8d7bb4 [term 1 LEADER]: Becoming Leader. State: Replica: ef66653e53ec49699727a771ac8d7bb4, State: Running, Role: LEADER
I20260620 18:38:11.762416 26265 consensus_queue.cc:237] T 8d42299735354b44b03cc58f6b02d6aa P ef66653e53ec49699727a771ac8d7bb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9e5bea8c365407db364bf7b7aebf653" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 38119 } } peers { permanent_uuid: "03685ee8dcd84c8db739f4a905afa6d8" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 41073 } } peers { permanent_uuid: "ef66653e53ec49699727a771ac8d7bb4" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 36169 } }
I20260620 18:38:11.766796 25755 catalog_manager.cc:5495] T 8d42299735354b44b03cc58f6b02d6aa P ef66653e53ec49699727a771ac8d7bb4 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef66653e53ec49699727a771ac8d7bb4 (127.23.115.2). New cstate: current_term: 1 leader_uuid: "ef66653e53ec49699727a771ac8d7bb4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9e5bea8c365407db364bf7b7aebf653" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 38119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "03685ee8dcd84c8db739f4a905afa6d8" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 41073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef66653e53ec49699727a771ac8d7bb4" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 36169 } health_report { overall_health: HEALTHY } } }
Jun 20 18:38:11 dist-test-slave-d645 krb5kdc[25710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980689, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.62@KRBTEST.COM for kudu/127.23.115.62@KRBTEST.COM
I20260620 18:38:11.822229 26277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:11.809499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:45559 (local address 127.0.0.1:45002)
0620 18:38:11.810058 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:11.810072 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:11.810210 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:11.810518 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:11.810523 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:11.811219 (+ 696us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:11.811503 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:11.811514 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:11.812476 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:11.812480 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:11.813632 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:11.813646 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:11.814354 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:11.814360 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:11.814501 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:11.815186 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:11.815203 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:11.817259 (+ 2056us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:11.819904 (+ 2645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:11.819915 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:11.819920 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:11.820326 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:11.820645 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:11.820648 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:11.820651 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:11.820854 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:11.821362 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:11.821368 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:11.821510 (+ 142us) client_negotiation.cc:769] Sending connection context
0620 18:38:11.821637 (+ 127us) client_negotiation.cc:240] Negotiation successful
0620 18:38:11.821869 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":306,"threads_started":1}
I20260620 18:38:11.822249 25850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:11.809434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45002 (local address 127.23.115.62:45559)
0620 18:38:11.809585 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:11.809589 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:11.810157 (+ 568us) server_negotiation.cc:407] Connection header received
0620 18:38:11.810317 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:11.810323 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:11.810397 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:11.810474 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:11.811650 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:11.812360 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:11.813769 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:11.814182 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:11.814291 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:11.817442 (+ 3151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:11.817446 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:11.817450 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:11.817480 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:11.819788 (+ 2308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:11.820427 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:11.820431 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:11.820433 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:11.820548 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:11.820963 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:11.820966 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:11.820968 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:11.821236 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:11.821345 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:11.821714 (+ 369us) server_negotiation.cc:299] Negotiation successful
0620 18:38:11.821998 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":101,"server-negotiator.queue_time_us":48}
I20260620 18:38:11.826992 25755 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.23.115.62@KRBTEST.COM'} at 127.0.0.1:45002:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 18:38:11.828173 25755 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:38:11.835047 25892 tablet_service.cc:1467] Processing CreateTablet for tablet c993758a8fa14ea3970a7cb60a11b7f2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9fd6fdd4043048e4bc27bd9a13f4e87f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 18:38:11.835059 26031 tablet_service.cc:1467] Processing CreateTablet for tablet c993758a8fa14ea3970a7cb60a11b7f2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9fd6fdd4043048e4bc27bd9a13f4e87f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 18:38:11.835377 26170 tablet_service.cc:1467] Processing CreateTablet for tablet c993758a8fa14ea3970a7cb60a11b7f2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9fd6fdd4043048e4bc27bd9a13f4e87f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 18:38:11.836045 26031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c993758a8fa14ea3970a7cb60a11b7f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:11.836045 25892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c993758a8fa14ea3970a7cb60a11b7f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:11.836359 26170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c993758a8fa14ea3970a7cb60a11b7f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:11.843281 26260 tablet_bootstrap.cc:492] T c993758a8fa14ea3970a7cb60a11b7f2 P 03685ee8dcd84c8db739f4a905afa6d8: Bootstrap starting.
I20260620 18:38:11.843317 26261 tablet_bootstrap.cc:492] T c993758a8fa14ea3970a7cb60a11b7f2 P e9e5bea8c365407db364bf7b7aebf653: Bootstrap starting.
I20260620 18:38:11.844168 26259 tablet_bootstrap.cc:492] T c993758a8fa14ea3970a7cb60a11b7f2 P ef66653e53ec49699727a771ac8d7bb4: Bootstrap starting.
I20260620 18:38:11.844909 26261 tablet_bootstrap.cc:654] T c993758a8fa14ea3970a7cb60a11b7f2 P e9e5bea8c365407db364bf7b7aebf653: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:11.844909 26260 tablet_bootstrap.cc:654] T c993758a8fa14ea3970a7cb60a11b7f2 P 03685ee8dcd84c8db739f4a905afa6d8: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:11.845732 26259 tablet_bootstrap.cc:654] T c993758a8fa14ea3970a7cb60a11b7f2 P ef66653e53ec49699727a771ac8d7bb4: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:11.857111 26261 tablet_bootstrap.cc:492] T c993758a8fa14ea3970a7cb60a11b7f2 P e9e5bea8c365407db364bf7b7aebf653: No bootstrap required, opened a new log
I20260620 18:38:11.857298 26261 ts_tablet_manager.cc:1397] T c993758a8fa14ea3970a7cb60a11b7f2 P e9e5bea8c365407db364bf7b7aebf653: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.007s
I20260620 18:38:11.857457 26260 tablet_bootstrap.cc:492] T c993758a8fa14ea3970a7cb60a11b7f2 P 03685ee8dcd84c8db739f4a905afa6d8: No bootstrap required, opened a new log
I20260620 18:38:11.857622 26260 ts_tablet_manager.cc:1397] T c993758a8fa14ea3970a7cb60a11b7f2 P 03685ee8dcd84c8db739f4a905afa6d8: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.008s
I20260620 18:38:11.857147 26259 tablet_bootstrap.cc:492] T c993758a8fa14ea3970a7cb60a11b7f2 P ef66653e53ec49699727a771ac8d7bb4: No bootstrap required, opened a new log
I20260620 18:38:11.858520 26259 ts_tablet_manager.cc:1397] T c993758a8fa14ea3970a7cb60a11b7f2 P ef66653e53ec49699727a771ac8d7bb4: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.011s
I20260620 18:38:11.858439 26261 raft_consensus.cc:348] T c993758a8fa14ea3970a7cb60a11b7f2 P e9e5bea8c365407db364bf7b7aebf653 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03685ee8dcd84c8db739f4a905afa6d8" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 41073 } } peers { permanent_uuid: "ef66653e53ec49699727a771ac8d7bb4" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 36169 } } peers { permanent_uuid: "e9e5bea8c365407db364bf7b7aebf653" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 38119 } }
I20260620 18:38:11.858523 26260 raft_consensus.cc:348] T c993758a8fa14ea3970a7cb60a11b7f2 P 03685ee8dcd84c8db739f4a905afa6d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03685ee8dcd84c8db739f4a905afa6d8" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 41073 } } peers { permanent_uuid: "ef66653e53ec49699727a771ac8d7bb4" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 36169 } } peers { permanent_uuid: "e9e5bea8c365407db364bf7b7aebf653" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 38119 } }
I20260620 18:38:11.858719 26261 raft_consensus.cc:374] T c993758a8fa14ea3970a7cb60a11b7f2 P e9e5bea8c365407db364bf7b7aebf653 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:11.858783 26260 raft_consensus.cc:374] T c993758a8fa14ea3970a7cb60a11b7f2 P 03685ee8dcd84c8db739f4a905afa6d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:11.858819 26261 raft_consensus.cc:729] T c993758a8fa14ea3970a7cb60a11b7f2 P e9e5bea8c365407db364bf7b7aebf653 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9e5bea8c365407db364bf7b7aebf653, State: Initialized, Role: FOLLOWER
I20260620 18:38:11.858879 26260 raft_consensus.cc:729] T c993758a8fa14ea3970a7cb60a11b7f2 P 03685ee8dcd84c8db739f4a905afa6d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03685ee8dcd84c8db739f4a905afa6d8, State: Initialized, Role: FOLLOWER
I20260620 18:38:11.859179 26260 consensus_queue.cc:260] T c993758a8fa14ea3970a7cb60a11b7f2 P 03685ee8dcd84c8db739f4a905afa6d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03685ee8dcd84c8db739f4a905afa6d8" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 41073 } } peers { permanent_uuid: "ef66653e53ec49699727a771ac8d7bb4" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 36169 } } peers { permanent_uuid: "e9e5bea8c365407db364bf7b7aebf653" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 38119 } }
I20260620 18:38:11.859182 26261 consensus_queue.cc:260] T c993758a8fa14ea3970a7cb60a11b7f2 P e9e5bea8c365407db364bf7b7aebf653 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03685ee8dcd84c8db739f4a905afa6d8" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 41073 } } peers { permanent_uuid: "ef66653e53ec49699727a771ac8d7bb4" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 36169 } } peers { permanent_uuid: "e9e5bea8c365407db364bf7b7aebf653" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 38119 } }
I20260620 18:38:11.859345 26259 raft_consensus.cc:348] T c993758a8fa14ea3970a7cb60a11b7f2 P ef66653e53ec49699727a771ac8d7bb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03685ee8dcd84c8db739f4a905afa6d8" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 41073 } } peers { permanent_uuid: "ef66653e53ec49699727a771ac8d7bb4" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 36169 } } peers { permanent_uuid: "e9e5bea8c365407db364bf7b7aebf653" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 38119 } }
I20260620 18:38:11.859555 26259 raft_consensus.cc:374] T c993758a8fa14ea3970a7cb60a11b7f2 P ef66653e53ec49699727a771ac8d7bb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:11.859637 26259 raft_consensus.cc:729] T c993758a8fa14ea3970a7cb60a11b7f2 P ef66653e53ec49699727a771ac8d7bb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef66653e53ec49699727a771ac8d7bb4, State: Initialized, Role: FOLLOWER
I20260620 18:38:11.859848 26259 consensus_queue.cc:260] T c993758a8fa14ea3970a7cb60a11b7f2 P ef66653e53ec49699727a771ac8d7bb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03685ee8dcd84c8db739f4a905afa6d8" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 41073 } } peers { permanent_uuid: "ef66653e53ec49699727a771ac8d7bb4" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 36169 } } peers { permanent_uuid: "e9e5bea8c365407db364bf7b7aebf653" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 38119 } }
I20260620 18:38:11.859823 26266 tablet_replica.cc:439] T c993758a8fa14ea3970a7cb60a11b7f2 P 03685ee8dcd84c8db739f4a905afa6d8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03685ee8dcd84c8db739f4a905afa6d8" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 41073 } } peers { permanent_uuid: "ef66653e53ec49699727a771ac8d7bb4" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 36169 } } peers { permanent_uuid: "e9e5bea8c365407db364bf7b7aebf653" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 38119 } } }
I20260620 18:38:11.860289 26266 tablet_replica.cc:442] T c993758a8fa14ea3970a7cb60a11b7f2 P 03685ee8dcd84c8db739f4a905afa6d8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:38:11.860350 26265 tablet_replica.cc:439] T c993758a8fa14ea3970a7cb60a11b7f2 P ef66653e53ec49699727a771ac8d7bb4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03685ee8dcd84c8db739f4a905afa6d8" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 41073 } } peers { permanent_uuid: "ef66653e53ec49699727a771ac8d7bb4" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 36169 } } peers { permanent_uuid: "e9e5bea8c365407db364bf7b7aebf653" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 38119 } } }
I20260620 18:38:11.860627 26265 tablet_replica.cc:442] T c993758a8fa14ea3970a7cb60a11b7f2 P ef66653e53ec49699727a771ac8d7bb4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:38:11.860769 26261 ts_tablet_manager.cc:1428] T c993758a8fa14ea3970a7cb60a11b7f2 P e9e5bea8c365407db364bf7b7aebf653: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 18:38:11.860930 26267 tablet_replica.cc:439] T c993758a8fa14ea3970a7cb60a11b7f2 P e9e5bea8c365407db364bf7b7aebf653: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03685ee8dcd84c8db739f4a905afa6d8" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 41073 } } peers { permanent_uuid: "ef66653e53ec49699727a771ac8d7bb4" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 36169 } } peers { permanent_uuid: "e9e5bea8c365407db364bf7b7aebf653" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 38119 } } }
I20260620 18:38:11.861265 26267 tablet_replica.cc:442] T c993758a8fa14ea3970a7cb60a11b7f2 P e9e5bea8c365407db364bf7b7aebf653: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:38:11.861896 26259 ts_tablet_manager.cc:1428] T c993758a8fa14ea3970a7cb60a11b7f2 P ef66653e53ec49699727a771ac8d7bb4: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260620 18:38:11.863245 26260 ts_tablet_manager.cc:1428] T c993758a8fa14ea3970a7cb60a11b7f2 P 03685ee8dcd84c8db739f4a905afa6d8: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260620 18:38:11.891534 26267 raft_consensus.cc:482] T c993758a8fa14ea3970a7cb60a11b7f2 P e9e5bea8c365407db364bf7b7aebf653 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:38:11.891762 26267 raft_consensus.cc:504] T c993758a8fa14ea3970a7cb60a11b7f2 P e9e5bea8c365407db364bf7b7aebf653 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03685ee8dcd84c8db739f4a905afa6d8" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 41073 } } peers { permanent_uuid: "ef66653e53ec49699727a771ac8d7bb4" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 36169 } } peers { permanent_uuid: "e9e5bea8c365407db364bf7b7aebf653" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 38119 } }
I20260620 18:38:11.893319 26267 leader_election.cc:288] T c993758a8fa14ea3970a7cb60a11b7f2 P e9e5bea8c365407db364bf7b7aebf653 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 03685ee8dcd84c8db739f4a905afa6d8 (127.23.115.1:41073), ef66653e53ec49699727a771ac8d7bb4 (127.23.115.2:36169)
I20260620 18:38:11.901942 26258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:11.894906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.3:36487 (local address 127.23.115.2:36169)
0620 18:38:11.895181 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:11.895187 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:11.895203 (+ 16us) server_negotiation.cc:407] Connection header received
0620 18:38:11.895285 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:11.895290 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:11.895358 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:11.895437 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:38:11.896051 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:11.897308 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:11.899689 (+ 2381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:11.900893 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:11.901058 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:11.901133 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:11.901545 (+ 412us) server_negotiation.cc:299] Negotiation successful
0620 18:38:11.901677 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148}
I20260620 18:38:11.902058 26281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:11.894078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.2:36169 (local address 127.23.115.3:36487)
0620 18:38:11.894578 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:11.894591 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:11.894676 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:11.895467 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:11.895471 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:11.895512 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:38:11.895923 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:11.895933 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:11.897434 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:11.897439 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:11.899535 (+ 2096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:11.899551 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:11.901109 (+ 1558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:11.901114 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:11.901301 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:11.901395 (+ 94us) client_negotiation.cc:769] Sending connection context
0620 18:38:11.901481 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 18:38:11.901658 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"mutex_wait_us":109,"thread_start_us":267,"threads_started":1}
I20260620 18:38:11.902014 26129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:11.893742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.1:41073 (local address 127.23.115.3:59287)
0620 18:38:11.893904 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:11.893921 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:11.894025 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:11.895369 (+ 1344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:11.895374 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:11.895399 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:38:11.895654 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:11.895667 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:11.897301 (+ 1634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:11.897306 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:11.899540 (+ 2234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:11.899551 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:11.901065 (+ 1514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:11.901070 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:11.901296 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:11.901388 (+ 92us) client_negotiation.cc:769] Sending connection context
0620 18:38:11.901481 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 18:38:11.901658 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":118}
I20260620 18:38:11.902343 26256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:11.893685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.3:59287 (local address 127.23.115.1:41073)
0620 18:38:11.893874 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:11.893880 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:11.895051 (+ 1171us) server_negotiation.cc:407] Connection header received
0620 18:38:11.895135 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:11.895140 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:11.895216 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:11.895333 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:38:11.895824 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:11.897144 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:11.899689 (+ 2545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:11.900902 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:11.901057 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:11.901133 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:11.901621 (+ 488us) server_negotiation.cc:299] Negotiation successful
0620 18:38:11.901751 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 18:38:11.902858 25912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c993758a8fa14ea3970a7cb60a11b7f2" candidate_uuid: "e9e5bea8c365407db364bf7b7aebf653" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03685ee8dcd84c8db739f4a905afa6d8" is_pre_election: true
I20260620 18:38:11.903115 25912 raft_consensus.cc:2454] T c993758a8fa14ea3970a7cb60a11b7f2 P 03685ee8dcd84c8db739f4a905afa6d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9e5bea8c365407db364bf7b7aebf653 in term 0.
I20260620 18:38:11.902971 26051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c993758a8fa14ea3970a7cb60a11b7f2" candidate_uuid: "e9e5bea8c365407db364bf7b7aebf653" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef66653e53ec49699727a771ac8d7bb4" is_pre_election: true
I20260620 18:38:11.903358 26051 raft_consensus.cc:2454] T c993758a8fa14ea3970a7cb60a11b7f2 P ef66653e53ec49699727a771ac8d7bb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9e5bea8c365407db364bf7b7aebf653 in term 0.
I20260620 18:38:11.903549 26121 leader_election.cc:302] T c993758a8fa14ea3970a7cb60a11b7f2 P e9e5bea8c365407db364bf7b7aebf653 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03685ee8dcd84c8db739f4a905afa6d8, e9e5bea8c365407db364bf7b7aebf653; no voters:
I20260620 18:38:11.903957 26267 raft_consensus.cc:2781] T c993758a8fa14ea3970a7cb60a11b7f2 P e9e5bea8c365407db364bf7b7aebf653 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:38:11.904093 26267 raft_consensus.cc:482] T c993758a8fa14ea3970a7cb60a11b7f2 P e9e5bea8c365407db364bf7b7aebf653 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:38:11.904212 26267 raft_consensus.cc:3037] T c993758a8fa14ea3970a7cb60a11b7f2 P e9e5bea8c365407db364bf7b7aebf653 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:11.906315 26267 raft_consensus.cc:504] T c993758a8fa14ea3970a7cb60a11b7f2 P e9e5bea8c365407db364bf7b7aebf653 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03685ee8dcd84c8db739f4a905afa6d8" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 41073 } } peers { permanent_uuid: "ef66653e53ec49699727a771ac8d7bb4" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 36169 } } peers { permanent_uuid: "e9e5bea8c365407db364bf7b7aebf653" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 38119 } }
I20260620 18:38:11.906821 26267 leader_election.cc:288] T c993758a8fa14ea3970a7cb60a11b7f2 P e9e5bea8c365407db364bf7b7aebf653 [CANDIDATE]: Term 1 election: Requested vote from peers 03685ee8dcd84c8db739f4a905afa6d8 (127.23.115.1:41073), ef66653e53ec49699727a771ac8d7bb4 (127.23.115.2:36169)
I20260620 18:38:11.907232 25912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c993758a8fa14ea3970a7cb60a11b7f2" candidate_uuid: "e9e5bea8c365407db364bf7b7aebf653" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03685ee8dcd84c8db739f4a905afa6d8"
I20260620 18:38:11.907321 26051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c993758a8fa14ea3970a7cb60a11b7f2" candidate_uuid: "e9e5bea8c365407db364bf7b7aebf653" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef66653e53ec49699727a771ac8d7bb4"
I20260620 18:38:11.907434 25912 raft_consensus.cc:3037] T c993758a8fa14ea3970a7cb60a11b7f2 P 03685ee8dcd84c8db739f4a905afa6d8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:11.907493 26051 raft_consensus.cc:3037] T c993758a8fa14ea3970a7cb60a11b7f2 P ef66653e53ec49699727a771ac8d7bb4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:11.909462 25912 raft_consensus.cc:2454] T c993758a8fa14ea3970a7cb60a11b7f2 P 03685ee8dcd84c8db739f4a905afa6d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9e5bea8c365407db364bf7b7aebf653 in term 1.
I20260620 18:38:11.909554 26051 raft_consensus.cc:2454] T c993758a8fa14ea3970a7cb60a11b7f2 P ef66653e53ec49699727a771ac8d7bb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9e5bea8c365407db364bf7b7aebf653 in term 1.
I20260620 18:38:11.909790 26121 leader_election.cc:302] T c993758a8fa14ea3970a7cb60a11b7f2 P e9e5bea8c365407db364bf7b7aebf653 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03685ee8dcd84c8db739f4a905afa6d8, e9e5bea8c365407db364bf7b7aebf653; no voters:
I20260620 18:38:11.910017 26267 raft_consensus.cc:2781] T c993758a8fa14ea3970a7cb60a11b7f2 P e9e5bea8c365407db364bf7b7aebf653 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:38:11.910413 26267 raft_consensus.cc:686] T c993758a8fa14ea3970a7cb60a11b7f2 P e9e5bea8c365407db364bf7b7aebf653 [term 1 LEADER]: Becoming Leader. State: Replica: e9e5bea8c365407db364bf7b7aebf653, State: Running, Role: LEADER
I20260620 18:38:11.910833 26267 consensus_queue.cc:237] T c993758a8fa14ea3970a7cb60a11b7f2 P e9e5bea8c365407db364bf7b7aebf653 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03685ee8dcd84c8db739f4a905afa6d8" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 41073 } } peers { permanent_uuid: "ef66653e53ec49699727a771ac8d7bb4" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 36169 } } peers { permanent_uuid: "e9e5bea8c365407db364bf7b7aebf653" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 38119 } }
I20260620 18:38:11.914462 26282 tablet_replica.cc:439] T c993758a8fa14ea3970a7cb60a11b7f2 P e9e5bea8c365407db364bf7b7aebf653: TxnStatusTablet state changed. Reason: New leader e9e5bea8c365407db364bf7b7aebf653. Latest consensus state: current_term: 1 leader_uuid: "e9e5bea8c365407db364bf7b7aebf653" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03685ee8dcd84c8db739f4a905afa6d8" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 41073 } } peers { permanent_uuid: "ef66653e53ec49699727a771ac8d7bb4" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 36169 } } peers { permanent_uuid: "e9e5bea8c365407db364bf7b7aebf653" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 38119 } } }
I20260620 18:38:11.914834 26282 tablet_replica.cc:442] T c993758a8fa14ea3970a7cb60a11b7f2 P e9e5bea8c365407db364bf7b7aebf653: This TxnStatusTablet replica's current role is: LEADER
I20260620 18:38:11.915446 26284 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 18:38:11.915681 26284 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 18:38:11.915809 25754 catalog_manager.cc:5495] T c993758a8fa14ea3970a7cb60a11b7f2 P e9e5bea8c365407db364bf7b7aebf653 reported cstate change: term changed from 0 to 1, leader changed from <none> to e9e5bea8c365407db364bf7b7aebf653 (127.23.115.3). New cstate: current_term: 1 leader_uuid: "e9e5bea8c365407db364bf7b7aebf653" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03685ee8dcd84c8db739f4a905afa6d8" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 41073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef66653e53ec49699727a771ac8d7bb4" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 36169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9e5bea8c365407db364bf7b7aebf653" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 38119 } health_report { overall_health: HEALTHY } } }
I20260620 18:38:11.942766 26255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:11.936766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47006 (local address 127.23.115.3:38119)
0620 18:38:11.936947 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:11.936952 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:11.936973 (+ 21us) server_negotiation.cc:407] Connection header received
0620 18:38:11.937201 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:11.937210 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:11.937322 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:11.937473 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:38:11.937824 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:11.939032 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:11.940263 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:11.940924 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:11.941048 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:11.941465 (+ 417us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:38:11.942127 (+ 662us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:38:11.942294 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:11.942365 (+ 71us) server_negotiation.cc:299] Negotiation successful
0620 18:38:11.942450 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 18:38:11.942899 26277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:11.936612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.3:38119 (local address 127.0.0.1:47006)
0620 18:38:11.936867 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:11.936884 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:11.936988 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:11.937491 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:11.937494 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:11.937504 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 18:38:11.937700 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:11.937708 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:11.939149 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:11.939153 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:11.940128 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:11.940140 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:11.941080 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:11.941084 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:11.941228 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:11.941287 (+ 59us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 18:38:11.942240 (+ 953us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 18:38:11.942260 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 18:38:11.942337 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 18:38:11.942472 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 18:38:11.950031 26051 raft_consensus.cc:1261] T c993758a8fa14ea3970a7cb60a11b7f2 P ef66653e53ec49699727a771ac8d7bb4 [term 1 FOLLOWER]: Refusing update from remote peer e9e5bea8c365407db364bf7b7aebf653: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:38:11.950281 25912 raft_consensus.cc:1261] T c993758a8fa14ea3970a7cb60a11b7f2 P 03685ee8dcd84c8db739f4a905afa6d8 [term 1 FOLLOWER]: Refusing update from remote peer e9e5bea8c365407db364bf7b7aebf653: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:38:11.950979 26282 consensus_queue.cc:1035] T c993758a8fa14ea3970a7cb60a11b7f2 P e9e5bea8c365407db364bf7b7aebf653 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef66653e53ec49699727a771ac8d7bb4" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 36169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:38:11.951305 26267 consensus_queue.cc:1035] T c993758a8fa14ea3970a7cb60a11b7f2 P e9e5bea8c365407db364bf7b7aebf653 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03685ee8dcd84c8db739f4a905afa6d8" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 41073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:38:11.961571 26265 tablet_replica.cc:439] T c993758a8fa14ea3970a7cb60a11b7f2 P ef66653e53ec49699727a771ac8d7bb4: TxnStatusTablet state changed. Reason: New leader e9e5bea8c365407db364bf7b7aebf653. Latest consensus state: current_term: 1 leader_uuid: "e9e5bea8c365407db364bf7b7aebf653" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03685ee8dcd84c8db739f4a905afa6d8" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 41073 } } peers { permanent_uuid: "ef66653e53ec49699727a771ac8d7bb4" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 36169 } } peers { permanent_uuid: "e9e5bea8c365407db364bf7b7aebf653" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 38119 } } }
I20260620 18:38:11.961958 26265 tablet_replica.cc:442] T c993758a8fa14ea3970a7cb60a11b7f2 P ef66653e53ec49699727a771ac8d7bb4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:38:11.965327 26289 mvcc.cc:205] Tried to move back new op lower bound from 7298992914218627072 to 7298992914071310336. Current Snapshot: MvccSnapshot[applied={T|T < 7298992914218627072}]
I20260620 18:38:11.965668 26287 mvcc.cc:205] Tried to move back new op lower bound from 7298992914218627072 to 7298992914071310336. Current Snapshot: MvccSnapshot[applied={T|T < 7298992914218627072}]
I20260620 18:38:11.966229 26267 tablet_replica.cc:439] T c993758a8fa14ea3970a7cb60a11b7f2 P e9e5bea8c365407db364bf7b7aebf653: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e9e5bea8c365407db364bf7b7aebf653" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03685ee8dcd84c8db739f4a905afa6d8" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 41073 } } peers { permanent_uuid: "ef66653e53ec49699727a771ac8d7bb4" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 36169 } } peers { permanent_uuid: "e9e5bea8c365407db364bf7b7aebf653" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 38119 } } }
I20260620 18:38:11.966463 26267 tablet_replica.cc:442] T c993758a8fa14ea3970a7cb60a11b7f2 P e9e5bea8c365407db364bf7b7aebf653: This TxnStatusTablet replica's current role is: LEADER
I20260620 18:38:11.966813 26266 tablet_replica.cc:439] T c993758a8fa14ea3970a7cb60a11b7f2 P 03685ee8dcd84c8db739f4a905afa6d8: TxnStatusTablet state changed. Reason: New leader e9e5bea8c365407db364bf7b7aebf653. Latest consensus state: current_term: 1 leader_uuid: "e9e5bea8c365407db364bf7b7aebf653" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03685ee8dcd84c8db739f4a905afa6d8" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 41073 } } peers { permanent_uuid: "ef66653e53ec49699727a771ac8d7bb4" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 36169 } } peers { permanent_uuid: "e9e5bea8c365407db364bf7b7aebf653" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 38119 } } }
I20260620 18:38:11.967095 26266 tablet_replica.cc:442] T c993758a8fa14ea3970a7cb60a11b7f2 P 03685ee8dcd84c8db739f4a905afa6d8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:38:11.968997 26267 tablet_replica.cc:439] T c993758a8fa14ea3970a7cb60a11b7f2 P e9e5bea8c365407db364bf7b7aebf653: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e9e5bea8c365407db364bf7b7aebf653" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03685ee8dcd84c8db739f4a905afa6d8" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 41073 } } peers { permanent_uuid: "ef66653e53ec49699727a771ac8d7bb4" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 36169 } } peers { permanent_uuid: "e9e5bea8c365407db364bf7b7aebf653" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 38119 } } }
I20260620 18:38:11.969265 26267 tablet_replica.cc:442] T c993758a8fa14ea3970a7cb60a11b7f2 P e9e5bea8c365407db364bf7b7aebf653: This TxnStatusTablet replica's current role is: LEADER
I20260620 18:38:11.971859 26288 mvcc.cc:205] Tried to move back new op lower bound from 7298992914218627072 to 7298992914071310336. Current Snapshot: MvccSnapshot[applied={T|T < 7298992914218627072}]
I20260620 18:38:11.972682 26265 tablet_replica.cc:439] T c993758a8fa14ea3970a7cb60a11b7f2 P ef66653e53ec49699727a771ac8d7bb4: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e9e5bea8c365407db364bf7b7aebf653" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03685ee8dcd84c8db739f4a905afa6d8" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 41073 } } peers { permanent_uuid: "ef66653e53ec49699727a771ac8d7bb4" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 36169 } } peers { permanent_uuid: "e9e5bea8c365407db364bf7b7aebf653" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 38119 } } }
I20260620 18:38:11.972956 26265 tablet_replica.cc:442] T c993758a8fa14ea3970a7cb60a11b7f2 P ef66653e53ec49699727a771ac8d7bb4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:38:11.976073 26266 tablet_replica.cc:439] T c993758a8fa14ea3970a7cb60a11b7f2 P 03685ee8dcd84c8db739f4a905afa6d8: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e9e5bea8c365407db364bf7b7aebf653" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03685ee8dcd84c8db739f4a905afa6d8" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 41073 } } peers { permanent_uuid: "ef66653e53ec49699727a771ac8d7bb4" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 36169 } } peers { permanent_uuid: "e9e5bea8c365407db364bf7b7aebf653" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 38119 } } }
I20260620 18:38:11.976326 26266 tablet_replica.cc:442] T c993758a8fa14ea3970a7cb60a11b7f2 P 03685ee8dcd84c8db739f4a905afa6d8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:38:11.982065 26258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:11.975137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48300 (local address 127.23.115.2:36169)
0620 18:38:11.975318 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:11.975324 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:11.975342 (+ 18us) server_negotiation.cc:407] Connection header received
0620 18:38:11.975426 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:11.975430 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:11.975539 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:11.975690 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:38:11.976958 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:11.978111 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:11.979930 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:11.980384 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:11.980492 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:11.980940 (+ 448us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:38:11.981409 (+ 469us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:38:11.981530 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:11.981632 (+ 102us) server_negotiation.cc:299] Negotiation successful
0620 18:38:11.981713 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 18:38:11.999508 26255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:11.993268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.2:48069 (local address 127.23.115.3:38119)
0620 18:38:11.993505 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:11.993511 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:11.993626 (+ 115us) server_negotiation.cc:407] Connection header received
0620 18:38:11.993908 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:11.993916 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:11.994001 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:11.994085 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:38:11.994663 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:11.995630 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:11.997437 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:11.998025 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:11.998170 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:11.998772 (+ 602us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:38:11.998873 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:38:11.998988 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:11.999118 (+ 130us) server_negotiation.cc:299] Negotiation successful
0620 18:38:11.999198 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 18:38:11.999605 26298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:11.992912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.3:38119 (local address 127.23.115.2:48069)
0620 18:38:11.993526 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:11.993545 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:11.993758 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:11.994125 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:11.994130 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:11.994143 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 18:38:11.994502 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:11.994511 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:11.995810 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:11.995814 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:11.997294 (+ 1480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:11.997308 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:11.998231 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:11.998239 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:11.998369 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:11.998425 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 18:38:11.999004 (+ 579us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 18:38:11.999017 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 18:38:11.999089 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 18:38:11.999224 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":621,"threads_started":1}
I20260620 18:38:12.010488 25912 raft_consensus.cc:1261] T 8d42299735354b44b03cc58f6b02d6aa P 03685ee8dcd84c8db739f4a905afa6d8 [term 1 FOLLOWER]: Refusing update from remote peer ef66653e53ec49699727a771ac8d7bb4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:38:12.011036 26190 raft_consensus.cc:1261] T 8d42299735354b44b03cc58f6b02d6aa P e9e5bea8c365407db364bf7b7aebf653 [term 1 FOLLOWER]: Refusing update from remote peer ef66653e53ec49699727a771ac8d7bb4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:38:12.011418 26270 consensus_queue.cc:1035] T 8d42299735354b44b03cc58f6b02d6aa P ef66653e53ec49699727a771ac8d7bb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03685ee8dcd84c8db739f4a905afa6d8" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 41073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:38:12.011830 26265 consensus_queue.cc:1035] T 8d42299735354b44b03cc58f6b02d6aa P ef66653e53ec49699727a771ac8d7bb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9e5bea8c365407db364bf7b7aebf653" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 38119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:38:12.052403 26258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:12.044313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.3:34519 (local address 127.23.115.2:36169)
0620 18:38:12.044532 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:12.044539 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:12.045271 (+ 732us) server_negotiation.cc:407] Connection header received
0620 18:38:12.046212 (+ 941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:12.046218 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:12.046325 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:12.046424 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:38:12.047158 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:12.048258 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:12.050591 (+ 2333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:12.051150 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:12.051253 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:12.051776 (+ 523us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:38:12.051886 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:38:12.051963 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:12.052112 (+ 149us) server_negotiation.cc:299] Negotiation successful
0620 18:38:12.052182 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 18:38:12.052901 26302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:12.043960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.2:36169 (local address 127.23.115.3:34519)
0620 18:38:12.045142 (+ 1182us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:12.045166 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:12.045328 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:12.046585 (+ 1257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:12.046591 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:12.046605 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 18:38:12.046972 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:12.046984 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:12.048448 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:12.048454 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:12.050451 (+ 1997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:12.050469 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:12.051337 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:12.051343 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:12.051501 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:12.051585 (+ 84us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 18:38:12.051983 (+ 398us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 18:38:12.052000 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 18:38:12.052096 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 18:38:12.052253 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":932,"thread_start_us":710,"threads_started":1}
I20260620 18:38:12.100262 25753 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45000:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:38:12.100549 25753 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45000:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:38:12.104405 25753 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7e98328eb2b244639dc33b9f555b6803: Sending DeleteTablet for 3 replicas of tablet 8d42299735354b44b03cc58f6b02d6aa
I20260620 18:38:12.105602 25892 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d42299735354b44b03cc58f6b02d6aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:38:12 UTC) from {username='kudu', principal='kudu/127.23.115.62@KRBTEST.COM'} at 127.0.0.1:50466
I20260620 18:38:12.105609 26170 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d42299735354b44b03cc58f6b02d6aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:38:12 UTC) from {username='kudu', principal='kudu/127.23.115.62@KRBTEST.COM'} at 127.0.0.1:46998
I20260620 18:38:12.106760 26031 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d42299735354b44b03cc58f6b02d6aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:38:12 UTC) from {username='kudu', principal='kudu/127.23.115.62@KRBTEST.COM'} at 127.0.0.1:48288
I20260620 18:38:12.113720 26308 tablet_replica.cc:330] T 8d42299735354b44b03cc58f6b02d6aa P 03685ee8dcd84c8db739f4a905afa6d8: stopping tablet replica
I20260620 18:38:12.114045 25754 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45000:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 18:38:12.114228 26308 raft_consensus.cc:2229] T 8d42299735354b44b03cc58f6b02d6aa P 03685ee8dcd84c8db739f4a905afa6d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:38:12.114652 26308 raft_consensus.cc:2258] T 8d42299735354b44b03cc58f6b02d6aa P 03685ee8dcd84c8db739f4a905afa6d8 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260620 18:38:12.114720 25754 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:38:12.119349 26309 tablet_replica.cc:330] T 8d42299735354b44b03cc58f6b02d6aa P e9e5bea8c365407db364bf7b7aebf653: stopping tablet replica
I20260620 18:38:12.119771 26309 raft_consensus.cc:2229] T 8d42299735354b44b03cc58f6b02d6aa P e9e5bea8c365407db364bf7b7aebf653 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:38:12.120078 26309 raft_consensus.cc:2258] T 8d42299735354b44b03cc58f6b02d6aa P e9e5bea8c365407db364bf7b7aebf653 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:38:12.121884 26309 ts_tablet_manager.cc:1905] T 8d42299735354b44b03cc58f6b02d6aa P e9e5bea8c365407db364bf7b7aebf653: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:38:12.123078 26030 tablet_service.cc:1467] Processing CreateTablet for tablet ccad77cd280242c6b596f52b8ebd35f9 (DEFAULT_TABLE table=test-table [id=acbd49fbbf004eef95eba87a64c85bb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:38:12.123629 26030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccad77cd280242c6b596f52b8ebd35f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:12.123867 25892 tablet_service.cc:1467] Processing CreateTablet for tablet ccad77cd280242c6b596f52b8ebd35f9 (DEFAULT_TABLE table=test-table [id=acbd49fbbf004eef95eba87a64c85bb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:38:12.124284 25892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccad77cd280242c6b596f52b8ebd35f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:12.128751 26308 ts_tablet_manager.cc:1905] T 8d42299735354b44b03cc58f6b02d6aa P 03685ee8dcd84c8db739f4a905afa6d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:38:12.128763 26309 ts_tablet_manager.cc:1918] T 8d42299735354b44b03cc58f6b02d6aa P e9e5bea8c365407db364bf7b7aebf653: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:38:12.128907 26309 log.cc:1196] T 8d42299735354b44b03cc58f6b02d6aa P e9e5bea8c365407db364bf7b7aebf653: Deleting WAL directory at /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980676253068-24012-0/minicluster-data/ts-2/wal/wals/8d42299735354b44b03cc58f6b02d6aa
I20260620 18:38:12.129380 26309 ts_tablet_manager.cc:1939] T 8d42299735354b44b03cc58f6b02d6aa P e9e5bea8c365407db364bf7b7aebf653: Deleting consensus metadata
I20260620 18:38:12.130297 26259 tablet_bootstrap.cc:492] T ccad77cd280242c6b596f52b8ebd35f9 P ef66653e53ec49699727a771ac8d7bb4: Bootstrap starting.
I20260620 18:38:12.130968 25740 catalog_manager.cc:4841] TS e9e5bea8c365407db364bf7b7aebf653 (127.23.115.3:38119): tablet 8d42299735354b44b03cc58f6b02d6aa (table test-table [id=c74afcfde1844d398f501c92c1b837ef]) successfully deleted
I20260620 18:38:12.132005 26259 tablet_bootstrap.cc:654] T ccad77cd280242c6b596f52b8ebd35f9 P ef66653e53ec49699727a771ac8d7bb4: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:12.135624 26308 ts_tablet_manager.cc:1918] T 8d42299735354b44b03cc58f6b02d6aa P 03685ee8dcd84c8db739f4a905afa6d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:38:12.135782 26308 log.cc:1196] T 8d42299735354b44b03cc58f6b02d6aa P 03685ee8dcd84c8db739f4a905afa6d8: Deleting WAL directory at /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980676253068-24012-0/minicluster-data/ts-0/wal/wals/8d42299735354b44b03cc58f6b02d6aa
I20260620 18:38:12.136301 26308 ts_tablet_manager.cc:1939] T 8d42299735354b44b03cc58f6b02d6aa P 03685ee8dcd84c8db739f4a905afa6d8: Deleting consensus metadata
I20260620 18:38:12.136747 26170 tablet_service.cc:1467] Processing CreateTablet for tablet ccad77cd280242c6b596f52b8ebd35f9 (DEFAULT_TABLE table=test-table [id=acbd49fbbf004eef95eba87a64c85bb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:38:12.137182 26260 tablet_bootstrap.cc:492] T ccad77cd280242c6b596f52b8ebd35f9 P 03685ee8dcd84c8db739f4a905afa6d8: Bootstrap starting.
I20260620 18:38:12.137559 26170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccad77cd280242c6b596f52b8ebd35f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:12.138819 26260 tablet_bootstrap.cc:654] T ccad77cd280242c6b596f52b8ebd35f9 P 03685ee8dcd84c8db739f4a905afa6d8: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:12.139846 25739 catalog_manager.cc:4841] TS 03685ee8dcd84c8db739f4a905afa6d8 (127.23.115.1:41073): tablet 8d42299735354b44b03cc58f6b02d6aa (table test-table [id=c74afcfde1844d398f501c92c1b837ef]) successfully deleted
I20260620 18:38:12.147619 26310 tablet_replica.cc:330] T 8d42299735354b44b03cc58f6b02d6aa P ef66653e53ec49699727a771ac8d7bb4: stopping tablet replica
I20260620 18:38:12.147831 26260 tablet_bootstrap.cc:492] T ccad77cd280242c6b596f52b8ebd35f9 P 03685ee8dcd84c8db739f4a905afa6d8: No bootstrap required, opened a new log
I20260620 18:38:12.147996 26260 ts_tablet_manager.cc:1397] T ccad77cd280242c6b596f52b8ebd35f9 P 03685ee8dcd84c8db739f4a905afa6d8: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.003s
I20260620 18:38:12.148079 26310 raft_consensus.cc:2229] T 8d42299735354b44b03cc58f6b02d6aa P ef66653e53ec49699727a771ac8d7bb4 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:38:12.148598 26259 tablet_bootstrap.cc:492] T ccad77cd280242c6b596f52b8ebd35f9 P ef66653e53ec49699727a771ac8d7bb4: No bootstrap required, opened a new log
I20260620 18:38:12.148723 26310 raft_consensus.cc:2258] T 8d42299735354b44b03cc58f6b02d6aa P ef66653e53ec49699727a771ac8d7bb4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:38:12.148710 26260 raft_consensus.cc:348] T ccad77cd280242c6b596f52b8ebd35f9 P 03685ee8dcd84c8db739f4a905afa6d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03685ee8dcd84c8db739f4a905afa6d8" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 41073 } } peers { permanent_uuid: "ef66653e53ec49699727a771ac8d7bb4" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 36169 } } peers { permanent_uuid: "e9e5bea8c365407db364bf7b7aebf653" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 38119 } }
I20260620 18:38:12.148885 26259 ts_tablet_manager.cc:1397] T ccad77cd280242c6b596f52b8ebd35f9 P ef66653e53ec49699727a771ac8d7bb4: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.002s
I20260620 18:38:12.149748 26259 raft_consensus.cc:348] T ccad77cd280242c6b596f52b8ebd35f9 P ef66653e53ec49699727a771ac8d7bb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03685ee8dcd84c8db739f4a905afa6d8" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 41073 } } peers { permanent_uuid: "ef66653e53ec49699727a771ac8d7bb4" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 36169 } } peers { permanent_uuid: "e9e5bea8c365407db364bf7b7aebf653" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 38119 } }
I20260620 18:38:12.150038 26259 raft_consensus.cc:374] T ccad77cd280242c6b596f52b8ebd35f9 P ef66653e53ec49699727a771ac8d7bb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:12.150158 26259 raft_consensus.cc:729] T ccad77cd280242c6b596f52b8ebd35f9 P ef66653e53ec49699727a771ac8d7bb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef66653e53ec49699727a771ac8d7bb4, State: Initialized, Role: FOLLOWER
I20260620 18:38:12.148970 26260 raft_consensus.cc:374] T ccad77cd280242c6b596f52b8ebd35f9 P 03685ee8dcd84c8db739f4a905afa6d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:12.150413 26260 raft_consensus.cc:729] T ccad77cd280242c6b596f52b8ebd35f9 P 03685ee8dcd84c8db739f4a905afa6d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03685ee8dcd84c8db739f4a905afa6d8, State: Initialized, Role: FOLLOWER
I20260620 18:38:12.150563 26259 consensus_queue.cc:260] T ccad77cd280242c6b596f52b8ebd35f9 P ef66653e53ec49699727a771ac8d7bb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03685ee8dcd84c8db739f4a905afa6d8" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 41073 } } peers { permanent_uuid: "ef66653e53ec49699727a771ac8d7bb4" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 36169 } } peers { permanent_uuid: "e9e5bea8c365407db364bf7b7aebf653" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 38119 } }
I20260620 18:38:12.150696 26260 consensus_queue.cc:260] T ccad77cd280242c6b596f52b8ebd35f9 P 03685ee8dcd84c8db739f4a905afa6d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03685ee8dcd84c8db739f4a905afa6d8" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 41073 } } peers { permanent_uuid: "ef66653e53ec49699727a771ac8d7bb4" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 36169 } } peers { permanent_uuid: "e9e5bea8c365407db364bf7b7aebf653" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 38119 } }
I20260620 18:38:12.152354 26259 ts_tablet_manager.cc:1428] T ccad77cd280242c6b596f52b8ebd35f9 P ef66653e53ec49699727a771ac8d7bb4: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260620 18:38:12.154376 26260 ts_tablet_manager.cc:1428] T ccad77cd280242c6b596f52b8ebd35f9 P 03685ee8dcd84c8db739f4a905afa6d8: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260620 18:38:12.158578 26310 ts_tablet_manager.cc:1905] T 8d42299735354b44b03cc58f6b02d6aa P ef66653e53ec49699727a771ac8d7bb4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:38:12.159824 26261 tablet_bootstrap.cc:492] T ccad77cd280242c6b596f52b8ebd35f9 P e9e5bea8c365407db364bf7b7aebf653: Bootstrap starting.
I20260620 18:38:12.161463 26261 tablet_bootstrap.cc:654] T ccad77cd280242c6b596f52b8ebd35f9 P e9e5bea8c365407db364bf7b7aebf653: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:12.166487 26310 ts_tablet_manager.cc:1918] T 8d42299735354b44b03cc58f6b02d6aa P ef66653e53ec49699727a771ac8d7bb4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:38:12.166708 26310 log.cc:1196] T 8d42299735354b44b03cc58f6b02d6aa P ef66653e53ec49699727a771ac8d7bb4: Deleting WAL directory at /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980676253068-24012-0/minicluster-data/ts-1/wal/wals/8d42299735354b44b03cc58f6b02d6aa
I20260620 18:38:12.168191 26310 ts_tablet_manager.cc:1939] T 8d42299735354b44b03cc58f6b02d6aa P ef66653e53ec49699727a771ac8d7bb4: Deleting consensus metadata
I20260620 18:38:12.168565 26261 tablet_bootstrap.cc:492] T ccad77cd280242c6b596f52b8ebd35f9 P e9e5bea8c365407db364bf7b7aebf653: No bootstrap required, opened a new log
I20260620 18:38:12.168824 26261 ts_tablet_manager.cc:1397] T ccad77cd280242c6b596f52b8ebd35f9 P e9e5bea8c365407db364bf7b7aebf653: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.006s
I20260620 18:38:12.169632 26261 raft_consensus.cc:348] T ccad77cd280242c6b596f52b8ebd35f9 P e9e5bea8c365407db364bf7b7aebf653 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03685ee8dcd84c8db739f4a905afa6d8" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 41073 } } peers { permanent_uuid: "ef66653e53ec49699727a771ac8d7bb4" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 36169 } } peers { permanent_uuid: "e9e5bea8c365407db364bf7b7aebf653" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 38119 } }
I20260620 18:38:12.169929 25739 catalog_manager.cc:4841] TS ef66653e53ec49699727a771ac8d7bb4 (127.23.115.2:36169): tablet 8d42299735354b44b03cc58f6b02d6aa (table test-table [id=c74afcfde1844d398f501c92c1b837ef]) successfully deleted
I20260620 18:38:12.169991 26261 raft_consensus.cc:374] T ccad77cd280242c6b596f52b8ebd35f9 P e9e5bea8c365407db364bf7b7aebf653 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:12.170166 26261 raft_consensus.cc:729] T ccad77cd280242c6b596f52b8ebd35f9 P e9e5bea8c365407db364bf7b7aebf653 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9e5bea8c365407db364bf7b7aebf653, State: Initialized, Role: FOLLOWER
I20260620 18:38:12.170469 26261 consensus_queue.cc:260] T ccad77cd280242c6b596f52b8ebd35f9 P e9e5bea8c365407db364bf7b7aebf653 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03685ee8dcd84c8db739f4a905afa6d8" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 41073 } } peers { permanent_uuid: "ef66653e53ec49699727a771ac8d7bb4" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 36169 } } peers { permanent_uuid: "e9e5bea8c365407db364bf7b7aebf653" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 38119 } }
I20260620 18:38:12.174880 26261 ts_tablet_manager.cc:1428] T ccad77cd280242c6b596f52b8ebd35f9 P e9e5bea8c365407db364bf7b7aebf653: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20260620 18:38:12.254995 26306 raft_consensus.cc:482] T ccad77cd280242c6b596f52b8ebd35f9 P e9e5bea8c365407db364bf7b7aebf653 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:38:12.255347 26306 raft_consensus.cc:504] T ccad77cd280242c6b596f52b8ebd35f9 P e9e5bea8c365407db364bf7b7aebf653 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03685ee8dcd84c8db739f4a905afa6d8" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 41073 } } peers { permanent_uuid: "ef66653e53ec49699727a771ac8d7bb4" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 36169 } } peers { permanent_uuid: "e9e5bea8c365407db364bf7b7aebf653" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 38119 } }
I20260620 18:38:12.256300 26306 leader_election.cc:288] T ccad77cd280242c6b596f52b8ebd35f9 P e9e5bea8c365407db364bf7b7aebf653 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 03685ee8dcd84c8db739f4a905afa6d8 (127.23.115.1:41073), ef66653e53ec49699727a771ac8d7bb4 (127.23.115.2:36169)
I20260620 18:38:12.257177 26051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccad77cd280242c6b596f52b8ebd35f9" candidate_uuid: "e9e5bea8c365407db364bf7b7aebf653" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef66653e53ec49699727a771ac8d7bb4" is_pre_election: true
I20260620 18:38:12.257450 26051 raft_consensus.cc:2454] T ccad77cd280242c6b596f52b8ebd35f9 P ef66653e53ec49699727a771ac8d7bb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9e5bea8c365407db364bf7b7aebf653 in term 0.
I20260620 18:38:12.257432 25911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccad77cd280242c6b596f52b8ebd35f9" candidate_uuid: "e9e5bea8c365407db364bf7b7aebf653" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03685ee8dcd84c8db739f4a905afa6d8" is_pre_election: true
I20260620 18:38:12.257697 25911 raft_consensus.cc:2454] T ccad77cd280242c6b596f52b8ebd35f9 P 03685ee8dcd84c8db739f4a905afa6d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9e5bea8c365407db364bf7b7aebf653 in term 0.
I20260620 18:38:12.257969 26121 leader_election.cc:302] T ccad77cd280242c6b596f52b8ebd35f9 P e9e5bea8c365407db364bf7b7aebf653 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e9e5bea8c365407db364bf7b7aebf653, ef66653e53ec49699727a771ac8d7bb4; no voters:
I20260620 18:38:12.258423 26306 raft_consensus.cc:2781] T ccad77cd280242c6b596f52b8ebd35f9 P e9e5bea8c365407db364bf7b7aebf653 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:38:12.258562 26306 raft_consensus.cc:482] T ccad77cd280242c6b596f52b8ebd35f9 P e9e5bea8c365407db364bf7b7aebf653 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:38:12.258674 26306 raft_consensus.cc:3037] T ccad77cd280242c6b596f52b8ebd35f9 P e9e5bea8c365407db364bf7b7aebf653 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:12.261919 26306 raft_consensus.cc:504] T ccad77cd280242c6b596f52b8ebd35f9 P e9e5bea8c365407db364bf7b7aebf653 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03685ee8dcd84c8db739f4a905afa6d8" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 41073 } } peers { permanent_uuid: "ef66653e53ec49699727a771ac8d7bb4" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 36169 } } peers { permanent_uuid: "e9e5bea8c365407db364bf7b7aebf653" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 38119 } }
I20260620 18:38:12.262619 26306 leader_election.cc:288] T ccad77cd280242c6b596f52b8ebd35f9 P e9e5bea8c365407db364bf7b7aebf653 [CANDIDATE]: Term 1 election: Requested vote from peers 03685ee8dcd84c8db739f4a905afa6d8 (127.23.115.1:41073), ef66653e53ec49699727a771ac8d7bb4 (127.23.115.2:36169)
I20260620 18:38:12.263212 25911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccad77cd280242c6b596f52b8ebd35f9" candidate_uuid: "e9e5bea8c365407db364bf7b7aebf653" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03685ee8dcd84c8db739f4a905afa6d8"
I20260620 18:38:12.263242 26051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccad77cd280242c6b596f52b8ebd35f9" candidate_uuid: "e9e5bea8c365407db364bf7b7aebf653" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef66653e53ec49699727a771ac8d7bb4"
I20260620 18:38:12.263435 25911 raft_consensus.cc:3037] T ccad77cd280242c6b596f52b8ebd35f9 P 03685ee8dcd84c8db739f4a905afa6d8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:12.263469 26051 raft_consensus.cc:3037] T ccad77cd280242c6b596f52b8ebd35f9 P ef66653e53ec49699727a771ac8d7bb4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:12.266592 26051 raft_consensus.cc:2454] T ccad77cd280242c6b596f52b8ebd35f9 P ef66653e53ec49699727a771ac8d7bb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9e5bea8c365407db364bf7b7aebf653 in term 1.
I20260620 18:38:12.266583 25911 raft_consensus.cc:2454] T ccad77cd280242c6b596f52b8ebd35f9 P 03685ee8dcd84c8db739f4a905afa6d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9e5bea8c365407db364bf7b7aebf653 in term 1.
I20260620 18:38:12.267163 26121 leader_election.cc:302] T ccad77cd280242c6b596f52b8ebd35f9 P e9e5bea8c365407db364bf7b7aebf653 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03685ee8dcd84c8db739f4a905afa6d8, e9e5bea8c365407db364bf7b7aebf653; no voters:
I20260620 18:38:12.267874 26306 raft_consensus.cc:2781] T ccad77cd280242c6b596f52b8ebd35f9 P e9e5bea8c365407db364bf7b7aebf653 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:38:12.268040 26306 raft_consensus.cc:686] T ccad77cd280242c6b596f52b8ebd35f9 P e9e5bea8c365407db364bf7b7aebf653 [term 1 LEADER]: Becoming Leader. State: Replica: e9e5bea8c365407db364bf7b7aebf653, State: Running, Role: LEADER
I20260620 18:38:12.268332 26306 consensus_queue.cc:237] T ccad77cd280242c6b596f52b8ebd35f9 P e9e5bea8c365407db364bf7b7aebf653 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03685ee8dcd84c8db739f4a905afa6d8" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 41073 } } peers { permanent_uuid: "ef66653e53ec49699727a771ac8d7bb4" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 36169 } } peers { permanent_uuid: "e9e5bea8c365407db364bf7b7aebf653" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 38119 } }
I20260620 18:38:12.271590 25754 catalog_manager.cc:5495] T ccad77cd280242c6b596f52b8ebd35f9 P e9e5bea8c365407db364bf7b7aebf653 reported cstate change: term changed from 0 to 1, leader changed from <none> to e9e5bea8c365407db364bf7b7aebf653 (127.23.115.3). New cstate: current_term: 1 leader_uuid: "e9e5bea8c365407db364bf7b7aebf653" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03685ee8dcd84c8db739f4a905afa6d8" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 41073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef66653e53ec49699727a771ac8d7bb4" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 36169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9e5bea8c365407db364bf7b7aebf653" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 38119 } health_report { overall_health: HEALTHY } } }
I20260620 18:38:12.307700 26255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:12.301228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47008 (local address 127.23.115.3:38119)
0620 18:38:12.301401 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:12.301407 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:12.301619 (+ 212us) server_negotiation.cc:407] Connection header received
0620 18:38:12.301867 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:12.301872 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:12.301953 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:12.302051 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:38:12.302475 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:12.303692 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:12.305557 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:12.306243 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:12.306370 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:12.306842 (+ 472us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:38:12.307028 (+ 186us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:38:12.307127 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:12.307251 (+ 124us) server_negotiation.cc:299] Negotiation successful
0620 18:38:12.307330 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 18:38:12.324604 26302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:12.317559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.3:38119 (local address 127.23.115.3:42907)
0620 18:38:12.317988 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:12.318008 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:12.318121 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:12.318897 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:12.318901 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:12.318913 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 18:38:12.319101 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:12.319112 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:12.320554 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:12.320558 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:12.322106 (+ 1548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:12.322120 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:12.323442 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:12.323447 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:12.323611 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:12.323659 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 18:38:12.324088 (+ 429us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 18:38:12.324096 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 18:38:12.324161 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 18:38:12.324293 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286}
I20260620 18:38:12.325309 26255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:12.318326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.3:42907 (local address 127.23.115.3:38119)
0620 18:38:12.318490 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:12.318499 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:12.318516 (+ 17us) server_negotiation.cc:407] Connection header received
0620 18:38:12.318571 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:12.318575 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:12.318657 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:12.318757 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:38:12.319250 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:12.320380 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:12.322568 (+ 2188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:12.323224 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:12.323343 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:12.323846 (+ 503us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:38:12.323963 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:38:12.324050 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:12.324996 (+ 946us) server_negotiation.cc:299] Negotiation successful
0620 18:38:12.325069 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 18:38:12.335336 25911 raft_consensus.cc:1261] T ccad77cd280242c6b596f52b8ebd35f9 P 03685ee8dcd84c8db739f4a905afa6d8 [term 1 FOLLOWER]: Refusing update from remote peer e9e5bea8c365407db364bf7b7aebf653: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:38:12.335366 26051 raft_consensus.cc:1261] T ccad77cd280242c6b596f52b8ebd35f9 P ef66653e53ec49699727a771ac8d7bb4 [term 1 FOLLOWER]: Refusing update from remote peer e9e5bea8c365407db364bf7b7aebf653: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:38:12.335981 26306 consensus_queue.cc:1035] T ccad77cd280242c6b596f52b8ebd35f9 P e9e5bea8c365407db364bf7b7aebf653 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03685ee8dcd84c8db739f4a905afa6d8" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 41073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:38:12.336467 26305 consensus_queue.cc:1035] T ccad77cd280242c6b596f52b8ebd35f9 P e9e5bea8c365407db364bf7b7aebf653 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef66653e53ec49699727a771ac8d7bb4" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 36169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 18:38:12.370555 26287 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 18:38:13.968250 25754 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45000:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:38:13.968447 25754 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45000:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:38:13.972564 25754 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7e98328eb2b244639dc33b9f555b6803: Sending DeleteTablet for 3 replicas of tablet ccad77cd280242c6b596f52b8ebd35f9
I20260620 18:38:13.973309 25892 tablet_service.cc:1514] Processing DeleteTablet for tablet ccad77cd280242c6b596f52b8ebd35f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:38:13 UTC) from {username='kudu', principal='kudu/127.23.115.62@KRBTEST.COM'} at 127.0.0.1:50466
I20260620 18:38:13.973368 26031 tablet_service.cc:1514] Processing DeleteTablet for tablet ccad77cd280242c6b596f52b8ebd35f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:38:13 UTC) from {username='kudu', principal='kudu/127.23.115.62@KRBTEST.COM'} at 127.0.0.1:48288
I20260620 18:38:13.973348 26169 tablet_service.cc:1514] Processing DeleteTablet for tablet ccad77cd280242c6b596f52b8ebd35f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:38:13 UTC) from {username='kudu', principal='kudu/127.23.115.62@KRBTEST.COM'} at 127.0.0.1:46998
I20260620 18:38:13.990044 26340 tablet_replica.cc:330] T ccad77cd280242c6b596f52b8ebd35f9 P e9e5bea8c365407db364bf7b7aebf653: stopping tablet replica
I20260620 18:38:13.990379 26340 raft_consensus.cc:2229] T ccad77cd280242c6b596f52b8ebd35f9 P e9e5bea8c365407db364bf7b7aebf653 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:38:13.991012 26340 raft_consensus.cc:2258] T ccad77cd280242c6b596f52b8ebd35f9 P e9e5bea8c365407db364bf7b7aebf653 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:38:13.992096 26340 ts_tablet_manager.cc:1905] T ccad77cd280242c6b596f52b8ebd35f9 P e9e5bea8c365407db364bf7b7aebf653: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:38:13.999583 26340 ts_tablet_manager.cc:1918] T ccad77cd280242c6b596f52b8ebd35f9 P e9e5bea8c365407db364bf7b7aebf653: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 18:38:13.999768 26340 log.cc:1196] T ccad77cd280242c6b596f52b8ebd35f9 P e9e5bea8c365407db364bf7b7aebf653: Deleting WAL directory at /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980676253068-24012-0/minicluster-data/ts-2/wal/wals/ccad77cd280242c6b596f52b8ebd35f9
I20260620 18:38:14.000286 26340 ts_tablet_manager.cc:1939] T ccad77cd280242c6b596f52b8ebd35f9 P e9e5bea8c365407db364bf7b7aebf653: Deleting consensus metadata
I20260620 18:38:14.001405 25740 catalog_manager.cc:4841] TS e9e5bea8c365407db364bf7b7aebf653 (127.23.115.3:38119): tablet ccad77cd280242c6b596f52b8ebd35f9 (table test-table [id=acbd49fbbf004eef95eba87a64c85bb2]) successfully deleted
I20260620 18:38:14.003576 26338 tablet_replica.cc:330] T ccad77cd280242c6b596f52b8ebd35f9 P 03685ee8dcd84c8db739f4a905afa6d8: stopping tablet replica
I20260620 18:38:14.004200 26339 tablet_replica.cc:330] T ccad77cd280242c6b596f52b8ebd35f9 P ef66653e53ec49699727a771ac8d7bb4: stopping tablet replica
I20260620 18:38:14.004426 26338 raft_consensus.cc:2229] T ccad77cd280242c6b596f52b8ebd35f9 P 03685ee8dcd84c8db739f4a905afa6d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:38:14.004451 26339 raft_consensus.cc:2229] T ccad77cd280242c6b596f52b8ebd35f9 P ef66653e53ec49699727a771ac8d7bb4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:38:14.004695 26339 raft_consensus.cc:2258] T ccad77cd280242c6b596f52b8ebd35f9 P ef66653e53ec49699727a771ac8d7bb4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:38:14.004743 26338 raft_consensus.cc:2258] T ccad77cd280242c6b596f52b8ebd35f9 P 03685ee8dcd84c8db739f4a905afa6d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:38:14.005703 26339 ts_tablet_manager.cc:1905] T ccad77cd280242c6b596f52b8ebd35f9 P ef66653e53ec49699727a771ac8d7bb4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:38:14.005764 26338 ts_tablet_manager.cc:1905] T ccad77cd280242c6b596f52b8ebd35f9 P 03685ee8dcd84c8db739f4a905afa6d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:38:14.010475 26339 ts_tablet_manager.cc:1918] T ccad77cd280242c6b596f52b8ebd35f9 P ef66653e53ec49699727a771ac8d7bb4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 18:38:14.010604 26339 log.cc:1196] T ccad77cd280242c6b596f52b8ebd35f9 P ef66653e53ec49699727a771ac8d7bb4: Deleting WAL directory at /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980676253068-24012-0/minicluster-data/ts-1/wal/wals/ccad77cd280242c6b596f52b8ebd35f9
I20260620 18:38:14.011009 26339 ts_tablet_manager.cc:1939] T ccad77cd280242c6b596f52b8ebd35f9 P ef66653e53ec49699727a771ac8d7bb4: Deleting consensus metadata
I20260620 18:38:14.011993 25739 catalog_manager.cc:4841] TS ef66653e53ec49699727a771ac8d7bb4 (127.23.115.2:36169): tablet ccad77cd280242c6b596f52b8ebd35f9 (table test-table [id=acbd49fbbf004eef95eba87a64c85bb2]) successfully deleted
I20260620 18:38:14.012426 26338 ts_tablet_manager.cc:1918] T ccad77cd280242c6b596f52b8ebd35f9 P 03685ee8dcd84c8db739f4a905afa6d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 18:38:14.012570 26338 log.cc:1196] T ccad77cd280242c6b596f52b8ebd35f9 P 03685ee8dcd84c8db739f4a905afa6d8: Deleting WAL directory at /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980676253068-24012-0/minicluster-data/ts-0/wal/wals/ccad77cd280242c6b596f52b8ebd35f9
I20260620 18:38:14.013002 26338 ts_tablet_manager.cc:1939] T ccad77cd280242c6b596f52b8ebd35f9 P 03685ee8dcd84c8db739f4a905afa6d8: Deleting consensus metadata
Jun 20 18:38:14 dist-test-slave-d645 krb5kdc[25710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980694, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
I20260620 18:38:14.014005 25739 catalog_manager.cc:4841] TS 03685ee8dcd84c8db739f4a905afa6d8 (127.23.115.1:41073): tablet ccad77cd280242c6b596f52b8ebd35f9 (table test-table [id=acbd49fbbf004eef95eba87a64c85bb2]) successfully deleted
Jun 20 18:38:14 dist-test-slave-d645 krb5kdc[25710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980694, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.115.62@KRBTEST.COM
I20260620 18:38:14.034374 26348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:14.021032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45012 (local address 127.23.115.62:45559)
0620 18:38:14.021502 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:14.021507 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:14.021586 (+ 79us) server_negotiation.cc:407] Connection header received
0620 18:38:14.021731 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:14.021735 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:14.021794 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:14.021913 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:14.023172 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:14.023930 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:14.025104 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:14.025485 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:14.025579 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:14.029370 (+ 3791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:14.029375 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:14.029379 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:14.029412 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:14.031962 (+ 2550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:14.032609 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:14.032613 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:14.032615 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:14.032761 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:14.033142 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:14.033145 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:14.033147 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:14.033438 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:14.033610 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:14.033941 (+ 331us) server_negotiation.cc:299] Negotiation successful
0620 18:38:14.034110 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":253,"threads_started":1}
I20260620 18:38:14.037698 25754 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45012:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:38:14.038349 25754 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:38:14.044663 25892 tablet_service.cc:1467] Processing CreateTablet for tablet f67c1c0c9c78482ba464b4465b0f27c4 (DEFAULT_TABLE table=test-table [id=09e92d3bba454eb08d12a8267186aa26]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:38:14.044921 26031 tablet_service.cc:1467] Processing CreateTablet for tablet f67c1c0c9c78482ba464b4465b0f27c4 (DEFAULT_TABLE table=test-table [id=09e92d3bba454eb08d12a8267186aa26]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:38:14.045121 25892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f67c1c0c9c78482ba464b4465b0f27c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:14.045302 26031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f67c1c0c9c78482ba464b4465b0f27c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:14.047336 26169 tablet_service.cc:1467] Processing CreateTablet for tablet f67c1c0c9c78482ba464b4465b0f27c4 (DEFAULT_TABLE table=test-table [id=09e92d3bba454eb08d12a8267186aa26]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:38:14.047760 26169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f67c1c0c9c78482ba464b4465b0f27c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:14.050266 26349 tablet_bootstrap.cc:492] T f67c1c0c9c78482ba464b4465b0f27c4 P ef66653e53ec49699727a771ac8d7bb4: Bootstrap starting.
I20260620 18:38:14.051115 26350 tablet_bootstrap.cc:492] T f67c1c0c9c78482ba464b4465b0f27c4 P 03685ee8dcd84c8db739f4a905afa6d8: Bootstrap starting.
I20260620 18:38:14.052253 26349 tablet_bootstrap.cc:654] T f67c1c0c9c78482ba464b4465b0f27c4 P ef66653e53ec49699727a771ac8d7bb4: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:14.052815 26350 tablet_bootstrap.cc:654] T f67c1c0c9c78482ba464b4465b0f27c4 P 03685ee8dcd84c8db739f4a905afa6d8: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:14.055542 26352 tablet_bootstrap.cc:492] T f67c1c0c9c78482ba464b4465b0f27c4 P e9e5bea8c365407db364bf7b7aebf653: Bootstrap starting.
I20260620 18:38:14.057156 26352 tablet_bootstrap.cc:654] T f67c1c0c9c78482ba464b4465b0f27c4 P e9e5bea8c365407db364bf7b7aebf653: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:14.057288 26349 tablet_bootstrap.cc:492] T f67c1c0c9c78482ba464b4465b0f27c4 P ef66653e53ec49699727a771ac8d7bb4: No bootstrap required, opened a new log
I20260620 18:38:14.057443 26349 ts_tablet_manager.cc:1397] T f67c1c0c9c78482ba464b4465b0f27c4 P ef66653e53ec49699727a771ac8d7bb4: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20260620 18:38:14.057547 26350 tablet_bootstrap.cc:492] T f67c1c0c9c78482ba464b4465b0f27c4 P 03685ee8dcd84c8db739f4a905afa6d8: No bootstrap required, opened a new log
I20260620 18:38:14.057696 26350 ts_tablet_manager.cc:1397] T f67c1c0c9c78482ba464b4465b0f27c4 P 03685ee8dcd84c8db739f4a905afa6d8: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260620 18:38:14.058167 26349 raft_consensus.cc:348] T f67c1c0c9c78482ba464b4465b0f27c4 P ef66653e53ec49699727a771ac8d7bb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03685ee8dcd84c8db739f4a905afa6d8" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 41073 } } peers { permanent_uuid: "ef66653e53ec49699727a771ac8d7bb4" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 36169 } } peers { permanent_uuid: "e9e5bea8c365407db364bf7b7aebf653" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 38119 } }
I20260620 18:38:14.058454 26349 raft_consensus.cc:374] T f67c1c0c9c78482ba464b4465b0f27c4 P ef66653e53ec49699727a771ac8d7bb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:14.058543 26349 raft_consensus.cc:729] T f67c1c0c9c78482ba464b4465b0f27c4 P ef66653e53ec49699727a771ac8d7bb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef66653e53ec49699727a771ac8d7bb4, State: Initialized, Role: FOLLOWER
I20260620 18:38:14.058396 26350 raft_consensus.cc:348] T f67c1c0c9c78482ba464b4465b0f27c4 P 03685ee8dcd84c8db739f4a905afa6d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03685ee8dcd84c8db739f4a905afa6d8" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 41073 } } peers { permanent_uuid: "ef66653e53ec49699727a771ac8d7bb4" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 36169 } } peers { permanent_uuid: "e9e5bea8c365407db364bf7b7aebf653" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 38119 } }
I20260620 18:38:14.058737 26350 raft_consensus.cc:374] T f67c1c0c9c78482ba464b4465b0f27c4 P 03685ee8dcd84c8db739f4a905afa6d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:14.058825 26349 consensus_queue.cc:260] T f67c1c0c9c78482ba464b4465b0f27c4 P ef66653e53ec49699727a771ac8d7bb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03685ee8dcd84c8db739f4a905afa6d8" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 41073 } } peers { permanent_uuid: "ef66653e53ec49699727a771ac8d7bb4" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 36169 } } peers { permanent_uuid: "e9e5bea8c365407db364bf7b7aebf653" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 38119 } }
I20260620 18:38:14.059115 26350 raft_consensus.cc:729] T f67c1c0c9c78482ba464b4465b0f27c4 P 03685ee8dcd84c8db739f4a905afa6d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03685ee8dcd84c8db739f4a905afa6d8, State: Initialized, Role: FOLLOWER
I20260620 18:38:14.059334 26352 tablet_bootstrap.cc:492] T f67c1c0c9c78482ba464b4465b0f27c4 P e9e5bea8c365407db364bf7b7aebf653: No bootstrap required, opened a new log
I20260620 18:38:14.059468 26352 ts_tablet_manager.cc:1397] T f67c1c0c9c78482ba464b4465b0f27c4 P e9e5bea8c365407db364bf7b7aebf653: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260620 18:38:14.059393 26350 consensus_queue.cc:260] T f67c1c0c9c78482ba464b4465b0f27c4 P 03685ee8dcd84c8db739f4a905afa6d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03685ee8dcd84c8db739f4a905afa6d8" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 41073 } } peers { permanent_uuid: "ef66653e53ec49699727a771ac8d7bb4" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 36169 } } peers { permanent_uuid: "e9e5bea8c365407db364bf7b7aebf653" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 38119 } }
I20260620 18:38:14.060271 26352 raft_consensus.cc:348] T f67c1c0c9c78482ba464b4465b0f27c4 P e9e5bea8c365407db364bf7b7aebf653 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03685ee8dcd84c8db739f4a905afa6d8" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 41073 } } peers { permanent_uuid: "ef66653e53ec49699727a771ac8d7bb4" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 36169 } } peers { permanent_uuid: "e9e5bea8c365407db364bf7b7aebf653" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 38119 } }
I20260620 18:38:14.060513 26352 raft_consensus.cc:374] T f67c1c0c9c78482ba464b4465b0f27c4 P e9e5bea8c365407db364bf7b7aebf653 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:14.060604 26352 raft_consensus.cc:729] T f67c1c0c9c78482ba464b4465b0f27c4 P e9e5bea8c365407db364bf7b7aebf653 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9e5bea8c365407db364bf7b7aebf653, State: Initialized, Role: FOLLOWER
I20260620 18:38:14.060846 26352 consensus_queue.cc:260] T f67c1c0c9c78482ba464b4465b0f27c4 P e9e5bea8c365407db364bf7b7aebf653 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03685ee8dcd84c8db739f4a905afa6d8" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 41073 } } peers { permanent_uuid: "ef66653e53ec49699727a771ac8d7bb4" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 36169 } } peers { permanent_uuid: "e9e5bea8c365407db364bf7b7aebf653" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 38119 } }
I20260620 18:38:14.061429 26352 ts_tablet_manager.cc:1428] T f67c1c0c9c78482ba464b4465b0f27c4 P e9e5bea8c365407db364bf7b7aebf653: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 18:38:14.061945 26350 ts_tablet_manager.cc:1428] T f67c1c0c9c78482ba464b4465b0f27c4 P 03685ee8dcd84c8db739f4a905afa6d8: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260620 18:38:14.063218 26349 ts_tablet_manager.cc:1428] T f67c1c0c9c78482ba464b4465b0f27c4 P ef66653e53ec49699727a771ac8d7bb4: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260620 18:38:14.209044 26355 raft_consensus.cc:482] T f67c1c0c9c78482ba464b4465b0f27c4 P ef66653e53ec49699727a771ac8d7bb4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:38:14.209259 26355 raft_consensus.cc:504] T f67c1c0c9c78482ba464b4465b0f27c4 P ef66653e53ec49699727a771ac8d7bb4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03685ee8dcd84c8db739f4a905afa6d8" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 41073 } } peers { permanent_uuid: "ef66653e53ec49699727a771ac8d7bb4" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 36169 } } peers { permanent_uuid: "e9e5bea8c365407db364bf7b7aebf653" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 38119 } }
I20260620 18:38:14.209867 26355 leader_election.cc:288] T f67c1c0c9c78482ba464b4465b0f27c4 P ef66653e53ec49699727a771ac8d7bb4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 03685ee8dcd84c8db739f4a905afa6d8 (127.23.115.1:41073), e9e5bea8c365407db364bf7b7aebf653 (127.23.115.3:38119)
I20260620 18:38:14.210472 25911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f67c1c0c9c78482ba464b4465b0f27c4" candidate_uuid: "ef66653e53ec49699727a771ac8d7bb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03685ee8dcd84c8db739f4a905afa6d8" is_pre_election: true
I20260620 18:38:14.210492 26189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f67c1c0c9c78482ba464b4465b0f27c4" candidate_uuid: "ef66653e53ec49699727a771ac8d7bb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9e5bea8c365407db364bf7b7aebf653" is_pre_election: true
I20260620 18:38:14.210691 25911 raft_consensus.cc:2454] T f67c1c0c9c78482ba464b4465b0f27c4 P 03685ee8dcd84c8db739f4a905afa6d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef66653e53ec49699727a771ac8d7bb4 in term 0.
I20260620 18:38:14.210750 26189 raft_consensus.cc:2454] T f67c1c0c9c78482ba464b4465b0f27c4 P e9e5bea8c365407db364bf7b7aebf653 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef66653e53ec49699727a771ac8d7bb4 in term 0.
I20260620 18:38:14.211164 25982 leader_election.cc:302] T f67c1c0c9c78482ba464b4465b0f27c4 P ef66653e53ec49699727a771ac8d7bb4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03685ee8dcd84c8db739f4a905afa6d8, ef66653e53ec49699727a771ac8d7bb4; no voters:
I20260620 18:38:14.211473 26355 raft_consensus.cc:2781] T f67c1c0c9c78482ba464b4465b0f27c4 P ef66653e53ec49699727a771ac8d7bb4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:38:14.211581 26355 raft_consensus.cc:482] T f67c1c0c9c78482ba464b4465b0f27c4 P ef66653e53ec49699727a771ac8d7bb4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:38:14.211697 26355 raft_consensus.cc:3037] T f67c1c0c9c78482ba464b4465b0f27c4 P ef66653e53ec49699727a771ac8d7bb4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:14.213757 26355 raft_consensus.cc:504] T f67c1c0c9c78482ba464b4465b0f27c4 P ef66653e53ec49699727a771ac8d7bb4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03685ee8dcd84c8db739f4a905afa6d8" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 41073 } } peers { permanent_uuid: "ef66653e53ec49699727a771ac8d7bb4" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 36169 } } peers { permanent_uuid: "e9e5bea8c365407db364bf7b7aebf653" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 38119 } }
I20260620 18:38:14.214416 26355 leader_election.cc:288] T f67c1c0c9c78482ba464b4465b0f27c4 P ef66653e53ec49699727a771ac8d7bb4 [CANDIDATE]: Term 1 election: Requested vote from peers 03685ee8dcd84c8db739f4a905afa6d8 (127.23.115.1:41073), e9e5bea8c365407db364bf7b7aebf653 (127.23.115.3:38119)
I20260620 18:38:14.215008 25911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f67c1c0c9c78482ba464b4465b0f27c4" candidate_uuid: "ef66653e53ec49699727a771ac8d7bb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03685ee8dcd84c8db739f4a905afa6d8"
I20260620 18:38:14.215018 26189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f67c1c0c9c78482ba464b4465b0f27c4" candidate_uuid: "ef66653e53ec49699727a771ac8d7bb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9e5bea8c365407db364bf7b7aebf653"
I20260620 18:38:14.215334 25911 raft_consensus.cc:3037] T f67c1c0c9c78482ba464b4465b0f27c4 P 03685ee8dcd84c8db739f4a905afa6d8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:14.215334 26189 raft_consensus.cc:3037] T f67c1c0c9c78482ba464b4465b0f27c4 P e9e5bea8c365407db364bf7b7aebf653 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:14.218227 26189 raft_consensus.cc:2454] T f67c1c0c9c78482ba464b4465b0f27c4 P e9e5bea8c365407db364bf7b7aebf653 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef66653e53ec49699727a771ac8d7bb4 in term 1.
I20260620 18:38:14.218209 25911 raft_consensus.cc:2454] T f67c1c0c9c78482ba464b4465b0f27c4 P 03685ee8dcd84c8db739f4a905afa6d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef66653e53ec49699727a771ac8d7bb4 in term 1.
I20260620 18:38:14.218676 25983 leader_election.cc:302] T f67c1c0c9c78482ba464b4465b0f27c4 P ef66653e53ec49699727a771ac8d7bb4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 03685ee8dcd84c8db739f4a905afa6d8, e9e5bea8c365407db364bf7b7aebf653, ef66653e53ec49699727a771ac8d7bb4; no voters:
I20260620 18:38:14.219033 26355 raft_consensus.cc:2781] T f67c1c0c9c78482ba464b4465b0f27c4 P ef66653e53ec49699727a771ac8d7bb4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:38:14.219524 26355 raft_consensus.cc:686] T f67c1c0c9c78482ba464b4465b0f27c4 P ef66653e53ec49699727a771ac8d7bb4 [term 1 LEADER]: Becoming Leader. State: Replica: ef66653e53ec49699727a771ac8d7bb4, State: Running, Role: LEADER
I20260620 18:38:14.219836 26355 consensus_queue.cc:237] T f67c1c0c9c78482ba464b4465b0f27c4 P ef66653e53ec49699727a771ac8d7bb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03685ee8dcd84c8db739f4a905afa6d8" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 41073 } } peers { permanent_uuid: "ef66653e53ec49699727a771ac8d7bb4" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 36169 } } peers { permanent_uuid: "e9e5bea8c365407db364bf7b7aebf653" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 38119 } }
I20260620 18:38:14.222353 25754 catalog_manager.cc:5495] T f67c1c0c9c78482ba464b4465b0f27c4 P ef66653e53ec49699727a771ac8d7bb4 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef66653e53ec49699727a771ac8d7bb4 (127.23.115.2). New cstate: current_term: 1 leader_uuid: "ef66653e53ec49699727a771ac8d7bb4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03685ee8dcd84c8db739f4a905afa6d8" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 41073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef66653e53ec49699727a771ac8d7bb4" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 36169 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9e5bea8c365407db364bf7b7aebf653" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 38119 } health_report { overall_health: UNKNOWN } } }
I20260620 18:38:14.262389 26360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:14.256359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48306 (local address 127.23.115.2:36169)
0620 18:38:14.257136 (+ 777us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:14.257143 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:14.257168 (+ 25us) server_negotiation.cc:407] Connection header received
0620 18:38:14.257248 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:14.257252 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:14.257329 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:14.257431 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:38:14.257888 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:14.258769 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:14.260529 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:14.261010 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:14.261133 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:14.261582 (+ 449us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:38:14.261709 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:38:14.261786 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:14.261942 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 18:38:14.262031 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":622,"thread_start_us":373,"threads_started":1}
I20260620 18:38:14.272413 25911 raft_consensus.cc:1261] T f67c1c0c9c78482ba464b4465b0f27c4 P 03685ee8dcd84c8db739f4a905afa6d8 [term 1 FOLLOWER]: Refusing update from remote peer ef66653e53ec49699727a771ac8d7bb4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:38:14.272437 26189 raft_consensus.cc:1261] T f67c1c0c9c78482ba464b4465b0f27c4 P e9e5bea8c365407db364bf7b7aebf653 [term 1 FOLLOWER]: Refusing update from remote peer ef66653e53ec49699727a771ac8d7bb4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:38:14.272946 26357 consensus_queue.cc:1035] T f67c1c0c9c78482ba464b4465b0f27c4 P ef66653e53ec49699727a771ac8d7bb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9e5bea8c365407db364bf7b7aebf653" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 38119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:38:14.273298 26355 consensus_queue.cc:1035] T f67c1c0c9c78482ba464b4465b0f27c4 P ef66653e53ec49699727a771ac8d7bb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03685ee8dcd84c8db739f4a905afa6d8" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 41073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:38:14.321125 25755 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45012:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:38:14.321352 25755 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45012:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:38:14.331215 25755 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7e98328eb2b244639dc33b9f555b6803: Sending DeleteTablet for 3 replicas of tablet f67c1c0c9c78482ba464b4465b0f27c4
I20260620 18:38:14.332109 25892 tablet_service.cc:1514] Processing DeleteTablet for tablet f67c1c0c9c78482ba464b4465b0f27c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:38:14 UTC) from {username='kudu', principal='kudu/127.23.115.62@KRBTEST.COM'} at 127.0.0.1:50466
I20260620 18:38:14.332157 26169 tablet_service.cc:1514] Processing DeleteTablet for tablet f67c1c0c9c78482ba464b4465b0f27c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:38:14 UTC) from {username='kudu', principal='kudu/127.23.115.62@KRBTEST.COM'} at 127.0.0.1:46998
I20260620 18:38:14.332244 26031 tablet_service.cc:1514] Processing DeleteTablet for tablet f67c1c0c9c78482ba464b4465b0f27c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:38:14 UTC) from {username='kudu', principal='kudu/127.23.115.62@KRBTEST.COM'} at 127.0.0.1:48288
I20260620 18:38:14.332435 26338 tablet_replica.cc:330] T f67c1c0c9c78482ba464b4465b0f27c4 P 03685ee8dcd84c8db739f4a905afa6d8: stopping tablet replica
I20260620 18:38:14.332523 26339 tablet_replica.cc:330] T f67c1c0c9c78482ba464b4465b0f27c4 P ef66653e53ec49699727a771ac8d7bb4: stopping tablet replica
I20260620 18:38:14.332640 26338 raft_consensus.cc:2229] T f67c1c0c9c78482ba464b4465b0f27c4 P 03685ee8dcd84c8db739f4a905afa6d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:38:14.332712 26339 raft_consensus.cc:2229] T f67c1c0c9c78482ba464b4465b0f27c4 P ef66653e53ec49699727a771ac8d7bb4 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:38:14.332850 26338 raft_consensus.cc:2258] T f67c1c0c9c78482ba464b4465b0f27c4 P 03685ee8dcd84c8db739f4a905afa6d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:38:14.333019 26339 raft_consensus.cc:2258] T f67c1c0c9c78482ba464b4465b0f27c4 P ef66653e53ec49699727a771ac8d7bb4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:38:14.333145 26340 tablet_replica.cc:330] T f67c1c0c9c78482ba464b4465b0f27c4 P e9e5bea8c365407db364bf7b7aebf653: stopping tablet replica
I20260620 18:38:14.333364 26340 raft_consensus.cc:2229] T f67c1c0c9c78482ba464b4465b0f27c4 P e9e5bea8c365407db364bf7b7aebf653 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:38:14.333582 26340 raft_consensus.cc:2258] T f67c1c0c9c78482ba464b4465b0f27c4 P e9e5bea8c365407db364bf7b7aebf653 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:38:14.333994 25755 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45012:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 18:38:14.334968 26339 ts_tablet_manager.cc:1905] T f67c1c0c9c78482ba464b4465b0f27c4 P ef66653e53ec49699727a771ac8d7bb4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:38:14.334991 26340 ts_tablet_manager.cc:1905] T f67c1c0c9c78482ba464b4465b0f27c4 P e9e5bea8c365407db364bf7b7aebf653: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:38:14.335222 26338 ts_tablet_manager.cc:1905] T f67c1c0c9c78482ba464b4465b0f27c4 P 03685ee8dcd84c8db739f4a905afa6d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:38:14.341567 26339 ts_tablet_manager.cc:1918] T f67c1c0c9c78482ba464b4465b0f27c4 P ef66653e53ec49699727a771ac8d7bb4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:38:14.341566 26340 ts_tablet_manager.cc:1918] T f67c1c0c9c78482ba464b4465b0f27c4 P e9e5bea8c365407db364bf7b7aebf653: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:38:14.341719 26340 log.cc:1196] T f67c1c0c9c78482ba464b4465b0f27c4 P e9e5bea8c365407db364bf7b7aebf653: Deleting WAL directory at /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980676253068-24012-0/minicluster-data/ts-2/wal/wals/f67c1c0c9c78482ba464b4465b0f27c4
I20260620 18:38:14.341817 26338 ts_tablet_manager.cc:1918] T f67c1c0c9c78482ba464b4465b0f27c4 P 03685ee8dcd84c8db739f4a905afa6d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:38:14.341982 26338 log.cc:1196] T f67c1c0c9c78482ba464b4465b0f27c4 P 03685ee8dcd84c8db739f4a905afa6d8: Deleting WAL directory at /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980676253068-24012-0/minicluster-data/ts-0/wal/wals/f67c1c0c9c78482ba464b4465b0f27c4
I20260620 18:38:14.342123 26340 ts_tablet_manager.cc:1939] T f67c1c0c9c78482ba464b4465b0f27c4 P e9e5bea8c365407db364bf7b7aebf653: Deleting consensus metadata
I20260620 18:38:14.342413 26338 ts_tablet_manager.cc:1939] T f67c1c0c9c78482ba464b4465b0f27c4 P 03685ee8dcd84c8db739f4a905afa6d8: Deleting consensus metadata
W20260620 18:38:14.342442 25755 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:38:14.343264 25740 catalog_manager.cc:4841] TS e9e5bea8c365407db364bf7b7aebf653 (127.23.115.3:38119): tablet f67c1c0c9c78482ba464b4465b0f27c4 (table test-table [id=09e92d3bba454eb08d12a8267186aa26]) successfully deleted
I20260620 18:38:14.343636 25739 catalog_manager.cc:4841] TS 03685ee8dcd84c8db739f4a905afa6d8 (127.23.115.1:41073): tablet f67c1c0c9c78482ba464b4465b0f27c4 (table test-table [id=09e92d3bba454eb08d12a8267186aa26]) successfully deleted
I20260620 18:38:14.344210 26339 log.cc:1196] T f67c1c0c9c78482ba464b4465b0f27c4 P ef66653e53ec49699727a771ac8d7bb4: Deleting WAL directory at /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980676253068-24012-0/minicluster-data/ts-1/wal/wals/f67c1c0c9c78482ba464b4465b0f27c4
I20260620 18:38:14.344627 26339 ts_tablet_manager.cc:1939] T f67c1c0c9c78482ba464b4465b0f27c4 P ef66653e53ec49699727a771ac8d7bb4: Deleting consensus metadata
I20260620 18:38:14.345468 25739 catalog_manager.cc:4841] TS ef66653e53ec49699727a771ac8d7bb4 (127.23.115.2:36169): tablet f67c1c0c9c78482ba464b4465b0f27c4 (table test-table [id=09e92d3bba454eb08d12a8267186aa26]) successfully deleted
I20260620 18:38:14.349313 25892 tablet_service.cc:1467] Processing CreateTablet for tablet 0b4612fb1dc946c98a40e3c4cb8eefdf (DEFAULT_TABLE table=test-table [id=38eaf8df1d3c4550a6497cc51a5968fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:38:14.349339 26031 tablet_service.cc:1467] Processing CreateTablet for tablet 0b4612fb1dc946c98a40e3c4cb8eefdf (DEFAULT_TABLE table=test-table [id=38eaf8df1d3c4550a6497cc51a5968fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:38:14.349423 26169 tablet_service.cc:1467] Processing CreateTablet for tablet 0b4612fb1dc946c98a40e3c4cb8eefdf (DEFAULT_TABLE table=test-table [id=38eaf8df1d3c4550a6497cc51a5968fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:38:14.349731 25892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b4612fb1dc946c98a40e3c4cb8eefdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:14.349773 26031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b4612fb1dc946c98a40e3c4cb8eefdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:14.349817 26169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b4612fb1dc946c98a40e3c4cb8eefdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:14.353883 26352 tablet_bootstrap.cc:492] T 0b4612fb1dc946c98a40e3c4cb8eefdf P e9e5bea8c365407db364bf7b7aebf653: Bootstrap starting.
I20260620 18:38:14.355438 26352 tablet_bootstrap.cc:654] T 0b4612fb1dc946c98a40e3c4cb8eefdf P e9e5bea8c365407db364bf7b7aebf653: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:14.355763 26350 tablet_bootstrap.cc:492] T 0b4612fb1dc946c98a40e3c4cb8eefdf P 03685ee8dcd84c8db739f4a905afa6d8: Bootstrap starting.
I20260620 18:38:14.356873 26349 tablet_bootstrap.cc:492] T 0b4612fb1dc946c98a40e3c4cb8eefdf P ef66653e53ec49699727a771ac8d7bb4: Bootstrap starting.
I20260620 18:38:14.357210 26352 tablet_bootstrap.cc:492] T 0b4612fb1dc946c98a40e3c4cb8eefdf P e9e5bea8c365407db364bf7b7aebf653: No bootstrap required, opened a new log
I20260620 18:38:14.357257 26350 tablet_bootstrap.cc:654] T 0b4612fb1dc946c98a40e3c4cb8eefdf P 03685ee8dcd84c8db739f4a905afa6d8: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:14.357358 26352 ts_tablet_manager.cc:1397] T 0b4612fb1dc946c98a40e3c4cb8eefdf P e9e5bea8c365407db364bf7b7aebf653: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260620 18:38:14.358083 26352 raft_consensus.cc:348] T 0b4612fb1dc946c98a40e3c4cb8eefdf P e9e5bea8c365407db364bf7b7aebf653 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03685ee8dcd84c8db739f4a905afa6d8" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 41073 } } peers { permanent_uuid: "ef66653e53ec49699727a771ac8d7bb4" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 36169 } } peers { permanent_uuid: "e9e5bea8c365407db364bf7b7aebf653" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 38119 } }
I20260620 18:38:14.358327 26349 tablet_bootstrap.cc:654] T 0b4612fb1dc946c98a40e3c4cb8eefdf P ef66653e53ec49699727a771ac8d7bb4: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:14.358316 26352 raft_consensus.cc:374] T 0b4612fb1dc946c98a40e3c4cb8eefdf P e9e5bea8c365407db364bf7b7aebf653 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:14.358388 26352 raft_consensus.cc:729] T 0b4612fb1dc946c98a40e3c4cb8eefdf P e9e5bea8c365407db364bf7b7aebf653 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9e5bea8c365407db364bf7b7aebf653, State: Initialized, Role: FOLLOWER
I20260620 18:38:14.358623 26352 consensus_queue.cc:260] T 0b4612fb1dc946c98a40e3c4cb8eefdf P e9e5bea8c365407db364bf7b7aebf653 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03685ee8dcd84c8db739f4a905afa6d8" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 41073 } } peers { permanent_uuid: "ef66653e53ec49699727a771ac8d7bb4" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 36169 } } peers { permanent_uuid: "e9e5bea8c365407db364bf7b7aebf653" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 38119 } }
I20260620 18:38:14.359444 26352 ts_tablet_manager.cc:1428] T 0b4612fb1dc946c98a40e3c4cb8eefdf P e9e5bea8c365407db364bf7b7aebf653: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 18:38:14.360105 26350 tablet_bootstrap.cc:492] T 0b4612fb1dc946c98a40e3c4cb8eefdf P 03685ee8dcd84c8db739f4a905afa6d8: No bootstrap required, opened a new log
I20260620 18:38:14.360273 26350 ts_tablet_manager.cc:1397] T 0b4612fb1dc946c98a40e3c4cb8eefdf P 03685ee8dcd84c8db739f4a905afa6d8: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 18:38:14.360296 26349 tablet_bootstrap.cc:492] T 0b4612fb1dc946c98a40e3c4cb8eefdf P ef66653e53ec49699727a771ac8d7bb4: No bootstrap required, opened a new log
I20260620 18:38:14.360435 26349 ts_tablet_manager.cc:1397] T 0b4612fb1dc946c98a40e3c4cb8eefdf P ef66653e53ec49699727a771ac8d7bb4: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260620 18:38:14.361162 26349 raft_consensus.cc:348] T 0b4612fb1dc946c98a40e3c4cb8eefdf P ef66653e53ec49699727a771ac8d7bb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03685ee8dcd84c8db739f4a905afa6d8" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 41073 } } peers { permanent_uuid: "ef66653e53ec49699727a771ac8d7bb4" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 36169 } } peers { permanent_uuid: "e9e5bea8c365407db364bf7b7aebf653" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 38119 } }
I20260620 18:38:14.361400 26349 raft_consensus.cc:374] T 0b4612fb1dc946c98a40e3c4cb8eefdf P ef66653e53ec49699727a771ac8d7bb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:14.361474 26349 raft_consensus.cc:729] T 0b4612fb1dc946c98a40e3c4cb8eefdf P ef66653e53ec49699727a771ac8d7bb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef66653e53ec49699727a771ac8d7bb4, State: Initialized, Role: FOLLOWER
I20260620 18:38:14.360962 26350 raft_consensus.cc:348] T 0b4612fb1dc946c98a40e3c4cb8eefdf P 03685ee8dcd84c8db739f4a905afa6d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03685ee8dcd84c8db739f4a905afa6d8" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 41073 } } peers { permanent_uuid: "ef66653e53ec49699727a771ac8d7bb4" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 36169 } } peers { permanent_uuid: "e9e5bea8c365407db364bf7b7aebf653" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 38119 } }
I20260620 18:38:14.361712 26350 raft_consensus.cc:374] T 0b4612fb1dc946c98a40e3c4cb8eefdf P 03685ee8dcd84c8db739f4a905afa6d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:14.361694 26349 consensus_queue.cc:260] T 0b4612fb1dc946c98a40e3c4cb8eefdf P ef66653e53ec49699727a771ac8d7bb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03685ee8dcd84c8db739f4a905afa6d8" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 41073 } } peers { permanent_uuid: "ef66653e53ec49699727a771ac8d7bb4" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 36169 } } peers { permanent_uuid: "e9e5bea8c365407db364bf7b7aebf653" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 38119 } }
I20260620 18:38:14.361802 26350 raft_consensus.cc:729] T 0b4612fb1dc946c98a40e3c4cb8eefdf P 03685ee8dcd84c8db739f4a905afa6d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03685ee8dcd84c8db739f4a905afa6d8, State: Initialized, Role: FOLLOWER
I20260620 18:38:14.362010 26350 consensus_queue.cc:260] T 0b4612fb1dc946c98a40e3c4cb8eefdf P 03685ee8dcd84c8db739f4a905afa6d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03685ee8dcd84c8db739f4a905afa6d8" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 41073 } } peers { permanent_uuid: "ef66653e53ec49699727a771ac8d7bb4" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 36169 } } peers { permanent_uuid: "e9e5bea8c365407db364bf7b7aebf653" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 38119 } }
I20260620 18:38:14.362304 26349 ts_tablet_manager.cc:1428] T 0b4612fb1dc946c98a40e3c4cb8eefdf P ef66653e53ec49699727a771ac8d7bb4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 18:38:14.363215 26350 ts_tablet_manager.cc:1428] T 0b4612fb1dc946c98a40e3c4cb8eefdf P 03685ee8dcd84c8db739f4a905afa6d8: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260620 18:38:14.483319 26355 raft_consensus.cc:482] T 0b4612fb1dc946c98a40e3c4cb8eefdf P ef66653e53ec49699727a771ac8d7bb4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:38:14.483548 26355 raft_consensus.cc:504] T 0b4612fb1dc946c98a40e3c4cb8eefdf P ef66653e53ec49699727a771ac8d7bb4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03685ee8dcd84c8db739f4a905afa6d8" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 41073 } } peers { permanent_uuid: "ef66653e53ec49699727a771ac8d7bb4" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 36169 } } peers { permanent_uuid: "e9e5bea8c365407db364bf7b7aebf653" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 38119 } }
I20260620 18:38:14.484187 26355 leader_election.cc:288] T 0b4612fb1dc946c98a40e3c4cb8eefdf P ef66653e53ec49699727a771ac8d7bb4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 03685ee8dcd84c8db739f4a905afa6d8 (127.23.115.1:41073), e9e5bea8c365407db364bf7b7aebf653 (127.23.115.3:38119)
I20260620 18:38:14.484658 25912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b4612fb1dc946c98a40e3c4cb8eefdf" candidate_uuid: "ef66653e53ec49699727a771ac8d7bb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03685ee8dcd84c8db739f4a905afa6d8" is_pre_election: true
I20260620 18:38:14.484652 26189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b4612fb1dc946c98a40e3c4cb8eefdf" candidate_uuid: "ef66653e53ec49699727a771ac8d7bb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9e5bea8c365407db364bf7b7aebf653" is_pre_election: true
I20260620 18:38:14.484876 25912 raft_consensus.cc:2454] T 0b4612fb1dc946c98a40e3c4cb8eefdf P 03685ee8dcd84c8db739f4a905afa6d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef66653e53ec49699727a771ac8d7bb4 in term 0.
I20260620 18:38:14.484947 26189 raft_consensus.cc:2454] T 0b4612fb1dc946c98a40e3c4cb8eefdf P e9e5bea8c365407db364bf7b7aebf653 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef66653e53ec49699727a771ac8d7bb4 in term 0.
I20260620 18:38:14.485234 25982 leader_election.cc:302] T 0b4612fb1dc946c98a40e3c4cb8eefdf P ef66653e53ec49699727a771ac8d7bb4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03685ee8dcd84c8db739f4a905afa6d8, ef66653e53ec49699727a771ac8d7bb4; no voters:
I20260620 18:38:14.485468 26355 raft_consensus.cc:2781] T 0b4612fb1dc946c98a40e3c4cb8eefdf P ef66653e53ec49699727a771ac8d7bb4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:38:14.485580 26355 raft_consensus.cc:482] T 0b4612fb1dc946c98a40e3c4cb8eefdf P ef66653e53ec49699727a771ac8d7bb4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:38:14.485684 26355 raft_consensus.cc:3037] T 0b4612fb1dc946c98a40e3c4cb8eefdf P ef66653e53ec49699727a771ac8d7bb4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:14.487977 26355 raft_consensus.cc:504] T 0b4612fb1dc946c98a40e3c4cb8eefdf P ef66653e53ec49699727a771ac8d7bb4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03685ee8dcd84c8db739f4a905afa6d8" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 41073 } } peers { permanent_uuid: "ef66653e53ec49699727a771ac8d7bb4" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 36169 } } peers { permanent_uuid: "e9e5bea8c365407db364bf7b7aebf653" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 38119 } }
I20260620 18:38:14.488494 26355 leader_election.cc:288] T 0b4612fb1dc946c98a40e3c4cb8eefdf P ef66653e53ec49699727a771ac8d7bb4 [CANDIDATE]: Term 1 election: Requested vote from peers 03685ee8dcd84c8db739f4a905afa6d8 (127.23.115.1:41073), e9e5bea8c365407db364bf7b7aebf653 (127.23.115.3:38119)
I20260620 18:38:14.488909 25912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b4612fb1dc946c98a40e3c4cb8eefdf" candidate_uuid: "ef66653e53ec49699727a771ac8d7bb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03685ee8dcd84c8db739f4a905afa6d8"
I20260620 18:38:14.488952 26189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b4612fb1dc946c98a40e3c4cb8eefdf" candidate_uuid: "ef66653e53ec49699727a771ac8d7bb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9e5bea8c365407db364bf7b7aebf653"
I20260620 18:38:14.489112 25912 raft_consensus.cc:3037] T 0b4612fb1dc946c98a40e3c4cb8eefdf P 03685ee8dcd84c8db739f4a905afa6d8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:14.489133 26189 raft_consensus.cc:3037] T 0b4612fb1dc946c98a40e3c4cb8eefdf P e9e5bea8c365407db364bf7b7aebf653 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:14.491117 25912 raft_consensus.cc:2454] T 0b4612fb1dc946c98a40e3c4cb8eefdf P 03685ee8dcd84c8db739f4a905afa6d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef66653e53ec49699727a771ac8d7bb4 in term 1.
I20260620 18:38:14.491542 25982 leader_election.cc:302] T 0b4612fb1dc946c98a40e3c4cb8eefdf P ef66653e53ec49699727a771ac8d7bb4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03685ee8dcd84c8db739f4a905afa6d8, ef66653e53ec49699727a771ac8d7bb4; no voters:
I20260620 18:38:14.492027 26189 raft_consensus.cc:2454] T 0b4612fb1dc946c98a40e3c4cb8eefdf P e9e5bea8c365407db364bf7b7aebf653 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef66653e53ec49699727a771ac8d7bb4 in term 1.
I20260620 18:38:14.492134 26355 raft_consensus.cc:2781] T 0b4612fb1dc946c98a40e3c4cb8eefdf P ef66653e53ec49699727a771ac8d7bb4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:38:14.492403 26355 raft_consensus.cc:686] T 0b4612fb1dc946c98a40e3c4cb8eefdf P ef66653e53ec49699727a771ac8d7bb4 [term 1 LEADER]: Becoming Leader. State: Replica: ef66653e53ec49699727a771ac8d7bb4, State: Running, Role: LEADER
I20260620 18:38:14.492756 26355 consensus_queue.cc:237] T 0b4612fb1dc946c98a40e3c4cb8eefdf P ef66653e53ec49699727a771ac8d7bb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03685ee8dcd84c8db739f4a905afa6d8" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 41073 } } peers { permanent_uuid: "ef66653e53ec49699727a771ac8d7bb4" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 36169 } } peers { permanent_uuid: "e9e5bea8c365407db364bf7b7aebf653" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 38119 } }
I20260620 18:38:14.495265 25755 catalog_manager.cc:5495] T 0b4612fb1dc946c98a40e3c4cb8eefdf P ef66653e53ec49699727a771ac8d7bb4 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef66653e53ec49699727a771ac8d7bb4 (127.23.115.2). New cstate: current_term: 1 leader_uuid: "ef66653e53ec49699727a771ac8d7bb4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03685ee8dcd84c8db739f4a905afa6d8" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 41073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef66653e53ec49699727a771ac8d7bb4" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 36169 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9e5bea8c365407db364bf7b7aebf653" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 38119 } health_report { overall_health: UNKNOWN } } }
I20260620 18:38:14.527933 26189 raft_consensus.cc:1261] T 0b4612fb1dc946c98a40e3c4cb8eefdf P e9e5bea8c365407db364bf7b7aebf653 [term 1 FOLLOWER]: Refusing update from remote peer ef66653e53ec49699727a771ac8d7bb4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:38:14.528098 25912 raft_consensus.cc:1261] T 0b4612fb1dc946c98a40e3c4cb8eefdf P 03685ee8dcd84c8db739f4a905afa6d8 [term 1 FOLLOWER]: Refusing update from remote peer ef66653e53ec49699727a771ac8d7bb4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:38:14.528448 26355 consensus_queue.cc:1035] T 0b4612fb1dc946c98a40e3c4cb8eefdf P ef66653e53ec49699727a771ac8d7bb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9e5bea8c365407db364bf7b7aebf653" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 38119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:38:14.528730 26357 consensus_queue.cc:1035] T 0b4612fb1dc946c98a40e3c4cb8eefdf P ef66653e53ec49699727a771ac8d7bb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03685ee8dcd84c8db739f4a905afa6d8" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 41073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 18:38:14.546615 26321 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 18:38:16.124591 25755 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45012:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:38:16.124799 25755 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45012:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:38:16.128801 25755 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7e98328eb2b244639dc33b9f555b6803: Sending DeleteTablet for 3 replicas of tablet 0b4612fb1dc946c98a40e3c4cb8eefdf
I20260620 18:38:16.129909 26031 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b4612fb1dc946c98a40e3c4cb8eefdf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:38:16 UTC) from {username='kudu', principal='kudu/127.23.115.62@KRBTEST.COM'} at 127.0.0.1:48288
I20260620 18:38:16.129951 25892 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b4612fb1dc946c98a40e3c4cb8eefdf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:38:16 UTC) from {username='kudu', principal='kudu/127.23.115.62@KRBTEST.COM'} at 127.0.0.1:50466
I20260620 18:38:16.130177 26169 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b4612fb1dc946c98a40e3c4cb8eefdf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:38:16 UTC) from {username='kudu', principal='kudu/127.23.115.62@KRBTEST.COM'} at 127.0.0.1:46998
I20260620 18:38:16.131244 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 25826
I20260620 18:38:16.143057 26396 tablet_replica.cc:330] T 0b4612fb1dc946c98a40e3c4cb8eefdf P ef66653e53ec49699727a771ac8d7bb4: stopping tablet replica
I20260620 18:38:16.143460 26396 raft_consensus.cc:2229] T 0b4612fb1dc946c98a40e3c4cb8eefdf P ef66653e53ec49699727a771ac8d7bb4 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:38:16.143975 26396 raft_consensus.cc:2258] T 0b4612fb1dc946c98a40e3c4cb8eefdf P ef66653e53ec49699727a771ac8d7bb4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:38:16.145586 26396 ts_tablet_manager.cc:1905] T 0b4612fb1dc946c98a40e3c4cb8eefdf P ef66653e53ec49699727a771ac8d7bb4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:38:16.146175 26397 tablet_replica.cc:330] T 0b4612fb1dc946c98a40e3c4cb8eefdf P e9e5bea8c365407db364bf7b7aebf653: stopping tablet replica
I20260620 18:38:16.146467 26397 raft_consensus.cc:2229] T 0b4612fb1dc946c98a40e3c4cb8eefdf P e9e5bea8c365407db364bf7b7aebf653 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:38:16.147439 26397 raft_consensus.cc:2258] T 0b4612fb1dc946c98a40e3c4cb8eefdf P e9e5bea8c365407db364bf7b7aebf653 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:38:16.147447 26395 tablet_replica.cc:330] T 0b4612fb1dc946c98a40e3c4cb8eefdf P 03685ee8dcd84c8db739f4a905afa6d8: stopping tablet replica
I20260620 18:38:16.147668 26395 raft_consensus.cc:2229] T 0b4612fb1dc946c98a40e3c4cb8eefdf P 03685ee8dcd84c8db739f4a905afa6d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:38:16.147980 26395 raft_consensus.cc:2258] T 0b4612fb1dc946c98a40e3c4cb8eefdf P 03685ee8dcd84c8db739f4a905afa6d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:38:16.148438 26397 ts_tablet_manager.cc:1905] T 0b4612fb1dc946c98a40e3c4cb8eefdf P e9e5bea8c365407db364bf7b7aebf653: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:38:16.149046 26395 ts_tablet_manager.cc:1905] T 0b4612fb1dc946c98a40e3c4cb8eefdf P 03685ee8dcd84c8db739f4a905afa6d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:38:16.153177 26396 ts_tablet_manager.cc:1918] T 0b4612fb1dc946c98a40e3c4cb8eefdf P ef66653e53ec49699727a771ac8d7bb4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 18:38:16.153374 26396 log.cc:1196] T 0b4612fb1dc946c98a40e3c4cb8eefdf P ef66653e53ec49699727a771ac8d7bb4: Deleting WAL directory at /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980676253068-24012-0/minicluster-data/ts-1/wal/wals/0b4612fb1dc946c98a40e3c4cb8eefdf
I20260620 18:38:16.153841 26396 ts_tablet_manager.cc:1939] T 0b4612fb1dc946c98a40e3c4cb8eefdf P ef66653e53ec49699727a771ac8d7bb4: Deleting consensus metadata
I20260620 18:38:16.154623 26397 ts_tablet_manager.cc:1918] T 0b4612fb1dc946c98a40e3c4cb8eefdf P e9e5bea8c365407db364bf7b7aebf653: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 18:38:16.154798 26397 log.cc:1196] T 0b4612fb1dc946c98a40e3c4cb8eefdf P e9e5bea8c365407db364bf7b7aebf653: Deleting WAL directory at /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980676253068-24012-0/minicluster-data/ts-2/wal/wals/0b4612fb1dc946c98a40e3c4cb8eefdf
I20260620 18:38:16.155083 25739 catalog_manager.cc:4841] TS ef66653e53ec49699727a771ac8d7bb4 (127.23.115.2:36169): tablet 0b4612fb1dc946c98a40e3c4cb8eefdf (table test-table [id=38eaf8df1d3c4550a6497cc51a5968fa]) successfully deleted
I20260620 18:38:16.155215 26397 ts_tablet_manager.cc:1939] T 0b4612fb1dc946c98a40e3c4cb8eefdf P e9e5bea8c365407db364bf7b7aebf653: Deleting consensus metadata
I20260620 18:38:16.155884 26395 ts_tablet_manager.cc:1918] T 0b4612fb1dc946c98a40e3c4cb8eefdf P 03685ee8dcd84c8db739f4a905afa6d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 18:38:16.156034 26395 log.cc:1196] T 0b4612fb1dc946c98a40e3c4cb8eefdf P 03685ee8dcd84c8db739f4a905afa6d8: Deleting WAL directory at /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980676253068-24012-0/minicluster-data/ts-0/wal/wals/0b4612fb1dc946c98a40e3c4cb8eefdf
I20260620 18:38:16.156103 25740 catalog_manager.cc:4841] TS e9e5bea8c365407db364bf7b7aebf653 (127.23.115.3:38119): tablet 0b4612fb1dc946c98a40e3c4cb8eefdf (table test-table [id=38eaf8df1d3c4550a6497cc51a5968fa]) successfully deleted
I20260620 18:38:16.156378 26395 ts_tablet_manager.cc:1939] T 0b4612fb1dc946c98a40e3c4cb8eefdf P 03685ee8dcd84c8db739f4a905afa6d8: Deleting consensus metadata
I20260620 18:38:16.157225 25739 catalog_manager.cc:4841] TS 03685ee8dcd84c8db739f4a905afa6d8 (127.23.115.1:41073): tablet 0b4612fb1dc946c98a40e3c4cb8eefdf (table test-table [id=38eaf8df1d3c4550a6497cc51a5968fa]) successfully deleted
Jun 20 18:38:16 dist-test-slave-d645 krb5kdc[25710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980694, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.115.1@KRBTEST.COM
I20260620 18:38:16.167824 26399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:16.132944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50476 (local address 127.23.115.1:41073)
0620 18:38:16.146476 (+ 13532us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:16.146484 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:16.147283 (+ 799us) server_negotiation.cc:407] Connection header received
0620 18:38:16.147361 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:16.147367 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:16.147463 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:16.147570 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:16.149516 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:16.150806 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:16.152768 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:16.153399 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:16.153530 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:16.162501 (+ 8971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:16.162517 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:16.162534 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:16.162568 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:16.165212 (+ 2644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:16.165855 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:16.165862 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:16.165864 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:16.165996 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:16.166346 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:16.166350 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:16.166352 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:16.166670 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:16.166975 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:16.167263 (+ 288us) server_negotiation.cc:299] Negotiation successful
0620 18:38:16.167488 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13350,"thread_start_us":13189,"threads_started":1}
W20260620 18:38:16.168430 25957 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50476
W20260620 18:38:16.169000 24012 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 25826: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:38:16.169198 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 25826
I20260620 18:38:16.184226 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 25966
Jun 20 18:38:16 dist-test-slave-d645 krb5kdc[25710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980694, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.115.2@KRBTEST.COM
I20260620 18:38:16.199501 26400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:16.185020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48318 (local address 127.23.115.2:36169)
0620 18:38:16.185528 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:16.185534 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:16.185555 (+ 21us) server_negotiation.cc:407] Connection header received
0620 18:38:16.185621 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:16.185626 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:16.185701 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:16.185806 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:16.187141 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:16.187949 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:16.189220 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:16.189602 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:16.189690 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:16.193945 (+ 4255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:16.193969 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:16.193989 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:16.194026 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:16.196954 (+ 2928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:16.197620 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:16.197627 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:16.197629 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:16.197765 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:16.198071 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:16.198075 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:16.198077 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:16.198471 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:16.198749 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:16.198965 (+ 216us) server_negotiation.cc:299] Negotiation successful
0620 18:38:16.199145 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":260,"threads_started":1}
W20260620 18:38:16.200177 26096 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48318
W20260620 18:38:16.200804 24012 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 25966: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:38:16.200958 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 25966
I20260620 18:38:16.218847 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 26105
Jun 20 18:38:16 dist-test-slave-d645 krb5kdc[25710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980694, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.115.3@KRBTEST.COM
I20260620 18:38:16.233881 26401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:16.219606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47020 (local address 127.23.115.3:38119)
0620 18:38:16.220078 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:16.220082 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:16.220098 (+ 16us) server_negotiation.cc:407] Connection header received
0620 18:38:16.220154 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:16.220158 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:16.220215 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:16.220309 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:16.221524 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:16.222309 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:16.223579 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:16.224036 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:16.224139 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:16.228353 (+ 4214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:16.228367 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:16.228384 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:16.228427 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:16.231344 (+ 2917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:16.231975 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:16.231983 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:16.231985 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:16.232107 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:16.232443 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:16.232447 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:16.232449 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:16.232783 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:16.233073 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:16.233331 (+ 258us) server_negotiation.cc:299] Negotiation successful
0620 18:38:16.233547 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":266,"threads_started":1}
W20260620 18:38:16.234508 26235 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47020
W20260620 18:38:16.235060 24012 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 26105: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:38:16.235200 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 26105
I20260620 18:38:16.256654 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 25723
I20260620 18:38:16.258381 25795 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:38:16.371593 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 25723
2026-06-20T18:38:16Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8113 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980676253068-24012-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:38:16 dist-test-slave-d645 krb5kdc[26405](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:38:16 dist-test-slave-d645 krb5kdc[26405](info): set up 2 sockets
Jun 20 18:38:16 dist-test-slave-d645 krb5kdc[26405](info): commencing operation
krb5kdc: starting...
W20260620 18:38:17.383153 24012 mini_kdc.cc:121] Time spent starting KDC: real 0.971s user 0.004s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980676253068-24012-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980676253068-24012-0/krb5kdc/test-user.keytab.
Jun 20 18:38:17 dist-test-slave-d645 krb5kdc[26405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980697, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:38:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:38:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.115.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.115.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.115.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab.
Entry for principal kudu/127.23.115.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab.
I20260620 18:38:17.556813 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980676253068-24012-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980676253068-24012-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980676253068-24012-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.115.62:40761
--webserver_interface=127.23.115.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab
--principal=kudu/127.23.115.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:37313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.115.62:40761
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:38:17.866946 26418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:17.867223 26418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:17.867358 26418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:17.874502 26418 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:38:17.874615 26418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:17.874671 26418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:17.874751 26418 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:38:17.874811 26418 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:38:17.884663 26418 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:37313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980676253068-24012-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980676253068-24012-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.115.62:40761
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab
--principal=kudu/127.23.115.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.115.62:40761
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.115.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980676253068-24012-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:38:17.885951 26418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:17.887243 26418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:17.898710 26424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:17.898746 26426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:17.898918 26423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:17.899166 26418 server_base.cc:1034] running on GCE node
I20260620 18:38:17.900061 26418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:17.901407 26418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:17.902634 26418 hybrid_clock.cc:648] HybridClock initialized: now 1781980697902594 us; error 28 us; skew 500 ppm
I20260620 18:38:17.903000 26418 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:38:17 dist-test-slave-d645 krb5kdc[26405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980697, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:17.905941 26418 init.cc:377] Logged in from keytab as kudu/127.23.115.62@KRBTEST.COM (short username kudu)
I20260620 18:38:17.907449 26418 webserver.cc:460] Webserver started at http://127.23.115.62:43935/ using document root <none> and password file <none>
I20260620 18:38:17.908089 26418 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:17.908185 26418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:17.908445 26418 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:38:17.911412 26418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980676253068-24012-0/minicluster-data/master-0/data/instance:
uuid: "93b7a3fa85304121a408a8d14ce48d63"
format_stamp: "Formatted at 2026-06-20 18:38:17 on dist-test-slave-d645"
server_key: "75b8a147be14a2861660731e47e8a0a1"
server_key_iv: "3750cc00d22fb9f15f512aac1ab1268b"
server_key_version: "encryptionkey@0"
I20260620 18:38:17.912125 26418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980676253068-24012-0/minicluster-data/master-0/wal/instance:
uuid: "93b7a3fa85304121a408a8d14ce48d63"
format_stamp: "Formatted at 2026-06-20 18:38:17 on dist-test-slave-d645"
server_key: "75b8a147be14a2861660731e47e8a0a1"
server_key_iv: "3750cc00d22fb9f15f512aac1ab1268b"
server_key_version: "encryptionkey@0"
I20260620 18:38:17.917415 26418 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260620 18:38:17.920913 26433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:17.922341 26418 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:38:17.922560 26418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980676253068-24012-0/minicluster-data/master-0/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980676253068-24012-0/minicluster-data/master-0/wal
uuid: "93b7a3fa85304121a408a8d14ce48d63"
format_stamp: "Formatted at 2026-06-20 18:38:17 on dist-test-slave-d645"
server_key: "75b8a147be14a2861660731e47e8a0a1"
server_key_iv: "3750cc00d22fb9f15f512aac1ab1268b"
server_key_version: "encryptionkey@0"
I20260620 18:38:17.922762 26418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980676253068-24012-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980676253068-24012-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980676253068-24012-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:17.940647 26418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:17.944733 26418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:17.944967 26418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:17.961603 26418 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.62:40761
I20260620 18:38:17.961611 26484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.62:40761 every 8 connection(s)
I20260620 18:38:17.962904 26418 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
I20260620 18:38:17.963517 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 26418
I20260620 18:38:17.963752 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980676253068-24012-0/minicluster-data/master-0/wal/instance
I20260620 18:38:17.964396 24012 external_mini_cluster.cc:1428] Setting key 5f928b6d943e88ac3c4a59346dc28a8b
I20260620 18:38:17.967216 26485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 18:38:17 dist-test-slave-d645 krb5kdc[26405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980697, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.115.62@KRBTEST.COM
I20260620 18:38:17.979204 26485 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93b7a3fa85304121a408a8d14ce48d63: Bootstrap starting.
I20260620 18:38:17.983057 26485 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 93b7a3fa85304121a408a8d14ce48d63: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:17.984221 26485 log.cc:824] T 00000000000000000000000000000000 P 93b7a3fa85304121a408a8d14ce48d63: Log is configured to *not* fsync() on all Append() calls
I20260620 18:38:17.987706 26485 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93b7a3fa85304121a408a8d14ce48d63: No bootstrap required, opened a new log
I20260620 18:38:17.988065 26488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:17.966354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38982 (local address 127.23.115.62:40761)
0620 18:38:17.967113 (+ 759us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:17.967128 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:17.967174 (+ 46us) server_negotiation.cc:407] Connection header received
0620 18:38:17.969362 (+ 2188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:17.969388 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:17.969842 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:17.970262 (+ 420us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:17.971690 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:17.973634 (+ 1944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:17.974893 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:17.975906 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:17.976083 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:17.979752 (+ 3669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:17.979762 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:17.979781 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:17.979824 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:17.983618 (+ 3794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:17.984317 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:17.984325 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:17.984332 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:17.984532 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:17.984931 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:17.984934 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:17.984937 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:17.985620 (+ 683us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:17.985915 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:17.986352 (+ 437us) server_negotiation.cc:299] Negotiation successful
0620 18:38:17.986771 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":320,"threads_started":1}
I20260620 18:38:17.995757 26485 raft_consensus.cc:348] T 00000000000000000000000000000000 P 93b7a3fa85304121a408a8d14ce48d63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93b7a3fa85304121a408a8d14ce48d63" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 40761 } }
I20260620 18:38:17.996100 26485 raft_consensus.cc:374] T 00000000000000000000000000000000 P 93b7a3fa85304121a408a8d14ce48d63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:17.996169 26485 raft_consensus.cc:729] T 00000000000000000000000000000000 P 93b7a3fa85304121a408a8d14ce48d63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93b7a3fa85304121a408a8d14ce48d63, State: Initialized, Role: FOLLOWER
I20260620 18:38:17.996682 26485 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93b7a3fa85304121a408a8d14ce48d63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93b7a3fa85304121a408a8d14ce48d63" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 40761 } }
I20260620 18:38:17.996954 26485 raft_consensus.cc:388] T 00000000000000000000000000000000 P 93b7a3fa85304121a408a8d14ce48d63 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:38:17.997090 26485 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93b7a3fa85304121a408a8d14ce48d63 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:38:17.997262 26485 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93b7a3fa85304121a408a8d14ce48d63 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:17.999393 26485 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93b7a3fa85304121a408a8d14ce48d63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93b7a3fa85304121a408a8d14ce48d63" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 40761 } }
I20260620 18:38:17.999848 26485 leader_election.cc:302] T 00000000000000000000000000000000 P 93b7a3fa85304121a408a8d14ce48d63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 93b7a3fa85304121a408a8d14ce48d63; no voters:
I20260620 18:38:18.000375 26485 leader_election.cc:288] T 00000000000000000000000000000000 P 93b7a3fa85304121a408a8d14ce48d63 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:38:18.000499 26490 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 93b7a3fa85304121a408a8d14ce48d63 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:38:18.001227 26490 raft_consensus.cc:686] T 00000000000000000000000000000000 P 93b7a3fa85304121a408a8d14ce48d63 [term 1 LEADER]: Becoming Leader. State: Replica: 93b7a3fa85304121a408a8d14ce48d63, State: Running, Role: LEADER
I20260620 18:38:18.001736 26490 consensus_queue.cc:237] T 00000000000000000000000000000000 P 93b7a3fa85304121a408a8d14ce48d63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93b7a3fa85304121a408a8d14ce48d63" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 40761 } }
I20260620 18:38:18.002427 26485 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93b7a3fa85304121a408a8d14ce48d63 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:38:18.004719 26492 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93b7a3fa85304121a408a8d14ce48d63 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93b7a3fa85304121a408a8d14ce48d63. Latest consensus state: current_term: 1 leader_uuid: "93b7a3fa85304121a408a8d14ce48d63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93b7a3fa85304121a408a8d14ce48d63" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 40761 } } }
I20260620 18:38:18.005136 26492 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93b7a3fa85304121a408a8d14ce48d63 [sys.catalog]: This master's current role is: LEADER
I20260620 18:38:18.005645 26491 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93b7a3fa85304121a408a8d14ce48d63 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "93b7a3fa85304121a408a8d14ce48d63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93b7a3fa85304121a408a8d14ce48d63" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 40761 } } }
I20260620 18:38:18.005955 26491 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93b7a3fa85304121a408a8d14ce48d63 [sys.catalog]: This master's current role is: LEADER
I20260620 18:38:18.007382 26497 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:38:18.011946 26497 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:38:18.021468 26497 catalog_manager.cc:1347] Generated new cluster ID: 81ac3b6687c142e4bf847ab887a0e06f
I20260620 18:38:18.021603 26497 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:38:18.047950 26497 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:38:18.049312 26497 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:38:18.058564 26497 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 93b7a3fa85304121a408a8d14ce48d63: Generated new TSK 0
I20260620 18:38:18.059428 26497 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.115.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.115.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.115.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.1.keytab.
Entry for principal kudu/127.23.115.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.1.keytab.
I20260620 18:38:18.107877 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980676253068-24012-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980676253068-24012-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980676253068-24012-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980676253068-24012-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.115.1:0
--local_ip_for_outbound_sockets=127.23.115.1
--webserver_interface=127.23.115.1
--webserver_port=0
--tserver_master_addrs=127.23.115.62:40761
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.1.keytab
--principal=kudu/127.23.115.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:37313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:38:18.426916 26511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:18.427197 26511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:18.427325 26511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:18.434433 26511 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:18.434537 26511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:18.434687 26511 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.115.1
I20260620 18:38:18.444130 26511 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:37313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980676253068-24012-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980676253068-24012-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.1.keytab
--principal=kudu/127.23.115.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.115.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980676253068-24012-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.115.1
--webserver_port=0
--tserver_master_addrs=127.23.115.62:40761
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.115.1
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980676253068-24012-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:38:18.445312 26511 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:18.446527 26511 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:18.458976 26516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:18.458923 26519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:18.458997 26517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:18.459695 26511 server_base.cc:1034] running on GCE node
I20260620 18:38:18.460259 26511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:18.461210 26511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:18.462440 26511 hybrid_clock.cc:648] HybridClock initialized: now 1781980698462383 us; error 47 us; skew 500 ppm
I20260620 18:38:18.462761 26511 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:38:18 dist-test-slave-d645 krb5kdc[26405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980698, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:18.465865 26511 init.cc:377] Logged in from keytab as kudu/127.23.115.1@KRBTEST.COM (short username kudu)
I20260620 18:38:18.467337 26511 webserver.cc:460] Webserver started at http://127.23.115.1:33563/ using document root <none> and password file <none>
I20260620 18:38:18.467998 26511 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:18.468098 26511 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:18.468376 26511 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:38:18.471573 26511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980676253068-24012-0/minicluster-data/ts-0/data/instance:
uuid: "5d3faaef9ee945b0a27080bc60d7a749"
format_stamp: "Formatted at 2026-06-20 18:38:18 on dist-test-slave-d645"
server_key: "be4b5f23b6acb39a5eb1cfac55592217"
server_key_iv: "fac96aea925e79426d29a04712422ffb"
server_key_version: "encryptionkey@0"
I20260620 18:38:18.472329 26511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980676253068-24012-0/minicluster-data/ts-0/wal/instance:
uuid: "5d3faaef9ee945b0a27080bc60d7a749"
format_stamp: "Formatted at 2026-06-20 18:38:18 on dist-test-slave-d645"
server_key: "be4b5f23b6acb39a5eb1cfac55592217"
server_key_iv: "fac96aea925e79426d29a04712422ffb"
server_key_version: "encryptionkey@0"
I20260620 18:38:18.477319 26511 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:38:18.480644 26526 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:18.481940 26511 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 18:38:18.482087 26511 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980676253068-24012-0/minicluster-data/ts-0/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980676253068-24012-0/minicluster-data/ts-0/wal
uuid: "5d3faaef9ee945b0a27080bc60d7a749"
format_stamp: "Formatted at 2026-06-20 18:38:18 on dist-test-slave-d645"
server_key: "be4b5f23b6acb39a5eb1cfac55592217"
server_key_iv: "fac96aea925e79426d29a04712422ffb"
server_key_version: "encryptionkey@0"
I20260620 18:38:18.482259 26511 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980676253068-24012-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980676253068-24012-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980676253068-24012-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:18.521512 26511 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:18.525718 26511 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:18.526039 26511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:18.527056 26511 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:38:18.528332 26511 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:38:18.528412 26511 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:18.528538 26511 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:38:18.528590 26511 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:18.559685 26511 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.1:40307
I20260620 18:38:18.559736 26638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.1:40307 every 8 connection(s)
I20260620 18:38:18.560856 26511 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980676253068-24012-0/minicluster-data/ts-0/data/info.pb
I20260620 18:38:18.566963 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 26511
I20260620 18:38:18.567166 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980676253068-24012-0/minicluster-data/ts-0/wal/instance
I20260620 18:38:18.567732 24012 external_mini_cluster.cc:1428] Setting key 946175099c8699b0749be5867f73083d
Jun 20 18:38:18 dist-test-slave-d645 krb5kdc[26405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980698, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.1@KRBTEST.COM for kudu/127.23.115.62@KRBTEST.COM
I20260620 18:38:18.580322 26641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:18.563165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.1:36821 (local address 127.23.115.62:40761)
0620 18:38:18.563547 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:18.563552 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:18.564463 (+ 911us) server_negotiation.cc:407] Connection header received
0620 18:38:18.566191 (+ 1728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:18.566196 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:18.566274 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:18.566367 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:18.567870 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:18.568679 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:18.570643 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:18.571117 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:18.571220 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:18.574919 (+ 3699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:18.574924 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:18.574927 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:18.574988 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:18.577458 (+ 2470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:18.578141 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:18.578147 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:18.578149 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:18.578272 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:18.578669 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:18.578674 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:18.578677 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:18.578927 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:18.579092 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:18.579749 (+ 657us) server_negotiation.cc:299] Negotiation successful
0620 18:38:18.579963 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":184,"threads_started":1}
I20260620 18:38:18.581228 26642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:18.563671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:40761 (local address 127.23.115.1:36821)
0620 18:38:18.564313 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:18.564354 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:18.565946 (+ 1592us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:18.566608 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:18.566615 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:18.567154 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:18.567711 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:18.567727 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:18.568843 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:18.568848 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:18.570490 (+ 1642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:18.570505 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:18.571330 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:18.571336 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:18.571624 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:18.572299 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:18.572325 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:18.574713 (+ 2388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:18.577598 (+ 2885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:18.577607 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:18.577621 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:18.578034 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:18.578374 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:18.578379 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:18.578382 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:18.578565 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:18.579082 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:18.579090 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:18.579399 (+ 309us) client_negotiation.cc:769] Sending connection context
0620 18:38:18.579636 (+ 237us) client_negotiation.cc:240] Negotiation successful
0620 18:38:18.580145 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":247,"threads_started":1}
I20260620 18:38:18.582685 26639 heartbeater.cc:347] Connected to a master server at 127.23.115.62:40761
I20260620 18:38:18.582996 26639 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.23.115.2@KRBTEST.COM; defaulting to no policy
I20260620 18:38:18.583694 26639 heartbeater.cc:511] Master 127.23.115.62:40761 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.115.2@KRBTEST.COM" created.
I20260620 18:38:18.585763 26450 ts_manager.cc:194] Registered new tserver with Master: 5d3faaef9ee945b0a27080bc60d7a749 (127.23.115.1:40307)
I20260620 18:38:18.587266 26450 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.115.1@KRBTEST.COM'} at 127.23.115.1:36821
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.115.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.2.keytab.
Entry for principal kudu/127.23.115.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.2.keytab.
I20260620 18:38:18.606251 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980676253068-24012-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980676253068-24012-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980676253068-24012-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980676253068-24012-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.115.2:0
--local_ip_for_outbound_sockets=127.23.115.2
--webserver_interface=127.23.115.2
--webserver_port=0
--tserver_master_addrs=127.23.115.62:40761
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.2.keytab
--principal=kudu/127.23.115.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:37313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:38:18.915899 26645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:18.916175 26645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:18.916267 26645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:18.923393 26645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:18.923506 26645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:18.923617 26645 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.115.2
I20260620 18:38:18.933807 26645 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:37313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980676253068-24012-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980676253068-24012-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.2.keytab
--principal=kudu/127.23.115.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.115.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980676253068-24012-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.115.2
--webserver_port=0
--tserver_master_addrs=127.23.115.62:40761
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.115.2
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980676253068-24012-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:38:18.935142 26645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:18.936450 26645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:18.949214 26650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:18.949265 26651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:18.949287 26653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:18.950119 26645 server_base.cc:1034] running on GCE node
I20260620 18:38:18.950672 26645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:18.951603 26645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:18.952818 26645 hybrid_clock.cc:648] HybridClock initialized: now 1781980698952770 us; error 35 us; skew 500 ppm
I20260620 18:38:18.953135 26645 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:38:18 dist-test-slave-d645 krb5kdc[26405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980698, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:18.956178 26645 init.cc:377] Logged in from keytab as kudu/127.23.115.2@KRBTEST.COM (short username kudu)
I20260620 18:38:18.957625 26645 webserver.cc:460] Webserver started at http://127.23.115.2:38371/ using document root <none> and password file <none>
I20260620 18:38:18.958256 26645 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:18.958356 26645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:18.958600 26645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:38:18.961580 26645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980676253068-24012-0/minicluster-data/ts-1/data/instance:
uuid: "96bd0a8f37be40b2be7f72f4185164e9"
format_stamp: "Formatted at 2026-06-20 18:38:18 on dist-test-slave-d645"
server_key: "b8da0620959472b16144254ce189a7a9"
server_key_iv: "231181f530f1ae5bcf40e4b59238620e"
server_key_version: "encryptionkey@0"
I20260620 18:38:18.962548 26645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980676253068-24012-0/minicluster-data/ts-1/wal/instance:
uuid: "96bd0a8f37be40b2be7f72f4185164e9"
format_stamp: "Formatted at 2026-06-20 18:38:18 on dist-test-slave-d645"
server_key: "b8da0620959472b16144254ce189a7a9"
server_key_iv: "231181f530f1ae5bcf40e4b59238620e"
server_key_version: "encryptionkey@0"
I20260620 18:38:18.968690 26645 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.001s
I20260620 18:38:18.972849 26660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:18.974314 26645 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 18:38:18.974466 26645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980676253068-24012-0/minicluster-data/ts-1/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980676253068-24012-0/minicluster-data/ts-1/wal
uuid: "96bd0a8f37be40b2be7f72f4185164e9"
format_stamp: "Formatted at 2026-06-20 18:38:18 on dist-test-slave-d645"
server_key: "b8da0620959472b16144254ce189a7a9"
server_key_iv: "231181f530f1ae5bcf40e4b59238620e"
server_key_version: "encryptionkey@0"
I20260620 18:38:18.974658 26645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980676253068-24012-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980676253068-24012-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980676253068-24012-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:19.004124 26645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:19.008340 26645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:19.008589 26645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:19.009690 26645 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:38:19.011030 26645 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:38:19.011113 26645 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:19.011227 26645 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:38:19.011284 26645 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:19.041563 26645 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.2:37977
I20260620 18:38:19.041610 26772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.2:37977 every 8 connection(s)
I20260620 18:38:19.042745 26645 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980676253068-24012-0/minicluster-data/ts-1/data/info.pb
I20260620 18:38:19.043201 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 26645
I20260620 18:38:19.043375 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980676253068-24012-0/minicluster-data/ts-1/wal/instance
I20260620 18:38:19.043872 24012 external_mini_cluster.cc:1428] Setting key 92f02c0abfbe589b4b6e0f66cba38d83
Jun 20 18:38:19 dist-test-slave-d645 krb5kdc[26405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980698, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.2@KRBTEST.COM for kudu/127.23.115.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.115.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.115.3@KRBTEST.COM" created.
I20260620 18:38:19.063009 26641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:19.045164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.2:34269 (local address 127.23.115.62:40761)
0620 18:38:19.045358 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:19.045365 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:19.046700 (+ 1335us) server_negotiation.cc:407] Connection header received
0620 18:38:19.048574 (+ 1874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:19.048579 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:19.048659 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:19.048758 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:19.050170 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:19.050875 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:19.053004 (+ 2129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:19.053439 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:19.053543 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:19.057437 (+ 3894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:19.057442 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:19.057445 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:19.057478 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:19.059908 (+ 2430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:19.060649 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:19.060653 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:19.060656 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:19.060785 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:19.061212 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:19.061216 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:19.061218 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:19.061541 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:19.061660 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:19.062402 (+ 742us) server_negotiation.cc:299] Negotiation successful
0620 18:38:19.062625 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 18:38:19.063858 26775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:19.045745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:40761 (local address 127.23.115.2:34269)
0620 18:38:19.046536 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:19.046580 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:19.048315 (+ 1735us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:19.048947 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:19.048955 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:19.049462 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:19.050006 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:19.050022 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:19.051037 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:19.051042 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:19.052847 (+ 1805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:19.052862 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:19.053668 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:19.053675 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:19.053932 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:19.054668 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:19.054693 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:19.057191 (+ 2498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:19.060054 (+ 2863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:19.060064 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:19.060083 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:19.060538 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:19.060888 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:19.060893 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:19.060897 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:19.061086 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:19.061656 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:19.061663 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:19.062020 (+ 357us) client_negotiation.cc:769] Sending connection context
0620 18:38:19.062284 (+ 264us) client_negotiation.cc:240] Negotiation successful
0620 18:38:19.062719 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":307,"threads_started":1}
I20260620 18:38:19.065201 26773 heartbeater.cc:347] Connected to a master server at 127.23.115.62:40761
I20260620 18:38:19.065532 26773 heartbeater.cc:464] Registering TS with master...
I20260620 18:38:19.066330 26773 heartbeater.cc:511] Master 127.23.115.62:40761 requested a full tablet report, sending...
I20260620 18:38:19.068356 26450 ts_manager.cc:194] Registered new tserver with Master: 96bd0a8f37be40b2be7f72f4185164e9 (127.23.115.2:37977)
I20260620 18:38:19.069648 26450 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.115.2@KRBTEST.COM'} at 127.23.115.2:34269
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.115.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.3.keytab.
Entry for principal kudu/127.23.115.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.3.keytab.
I20260620 18:38:19.085922 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980676253068-24012-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980676253068-24012-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980676253068-24012-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980676253068-24012-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.115.3:0
--local_ip_for_outbound_sockets=127.23.115.3
--webserver_interface=127.23.115.3
--webserver_port=0
--tserver_master_addrs=127.23.115.62:40761
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.3.keytab
--principal=kudu/127.23.115.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:37313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:38:19.393009 26778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:19.393318 26778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:19.393457 26778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:19.400686 26778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:19.400807 26778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:19.400949 26778 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.115.3
I20260620 18:38:19.410694 26778 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:37313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980676253068-24012-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980676253068-24012-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.3.keytab
--principal=kudu/127.23.115.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.115.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980676253068-24012-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.115.3
--webserver_port=0
--tserver_master_addrs=127.23.115.62:40761
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.115.3
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980676253068-24012-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:38:19.412006 26778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:19.413223 26778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:19.425453 26783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:19.425519 26786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:19.425498 26784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:19.426317 26778 server_base.cc:1034] running on GCE node
I20260620 18:38:19.426864 26778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:19.427829 26778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:19.429039 26778 hybrid_clock.cc:648] HybridClock initialized: now 1781980699428992 us; error 39 us; skew 500 ppm
I20260620 18:38:19.429308 26778 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:38:19 dist-test-slave-d645 krb5kdc[26405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980699, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:19.432251 26778 init.cc:377] Logged in from keytab as kudu/127.23.115.3@KRBTEST.COM (short username kudu)
I20260620 18:38:19.433563 26778 webserver.cc:460] Webserver started at http://127.23.115.3:44403/ using document root <none> and password file <none>
I20260620 18:38:19.434165 26778 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:19.434266 26778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:19.434506 26778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:38:19.437173 26778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980676253068-24012-0/minicluster-data/ts-2/data/instance:
uuid: "681a621c11ba4711ad7e61cac1898745"
format_stamp: "Formatted at 2026-06-20 18:38:19 on dist-test-slave-d645"
server_key: "1a2ec6b2d9a63c8b409f166e978aface"
server_key_iv: "1b21e4fc4b44614fd13ed7ac9d90ecfb"
server_key_version: "encryptionkey@0"
I20260620 18:38:19.437867 26778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980676253068-24012-0/minicluster-data/ts-2/wal/instance:
uuid: "681a621c11ba4711ad7e61cac1898745"
format_stamp: "Formatted at 2026-06-20 18:38:19 on dist-test-slave-d645"
server_key: "1a2ec6b2d9a63c8b409f166e978aface"
server_key_iv: "1b21e4fc4b44614fd13ed7ac9d90ecfb"
server_key_version: "encryptionkey@0"
I20260620 18:38:19.442571 26778 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260620 18:38:19.446028 26793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:19.447517 26778 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:38:19.447646 26778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980676253068-24012-0/minicluster-data/ts-2/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980676253068-24012-0/minicluster-data/ts-2/wal
uuid: "681a621c11ba4711ad7e61cac1898745"
format_stamp: "Formatted at 2026-06-20 18:38:19 on dist-test-slave-d645"
server_key: "1a2ec6b2d9a63c8b409f166e978aface"
server_key_iv: "1b21e4fc4b44614fd13ed7ac9d90ecfb"
server_key_version: "encryptionkey@0"
I20260620 18:38:19.447813 26778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980676253068-24012-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980676253068-24012-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980676253068-24012-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:19.461730 26778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:19.465764 26778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:19.466003 26778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:19.466868 26778 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:38:19.468171 26778 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:38:19.468247 26778 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:19.468334 26778 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:38:19.468379 26778 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:19.499615 26778 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.3:42271
I20260620 18:38:19.499759 26905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.3:42271 every 8 connection(s)
I20260620 18:38:19.500931 26778 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980676253068-24012-0/minicluster-data/ts-2/data/info.pb
I20260620 18:38:19.502648 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 26778
I20260620 18:38:19.502843 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980676253068-24012-0/minicluster-data/ts-2/wal/instance
I20260620 18:38:19.503360 24012 external_mini_cluster.cc:1428] Setting key 3004ec98f38c16a16ab53c44bda0d0e4
Jun 20 18:38:19 dist-test-slave-d645 krb5kdc[26405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980699, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.3@KRBTEST.COM for kudu/127.23.115.62@KRBTEST.COM
I20260620 18:38:19.521958 26641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:19.503875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.3:35083 (local address 127.23.115.62:40761)
0620 18:38:19.504091 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:19.504097 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:19.505368 (+ 1271us) server_negotiation.cc:407] Connection header received
0620 18:38:19.507364 (+ 1996us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:19.507370 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:19.507451 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:19.507535 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:19.509011 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:19.509976 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:19.511916 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:19.512377 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:19.512490 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:19.515833 (+ 3343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:19.515838 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:19.515842 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:19.515873 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:19.519070 (+ 3197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:19.519770 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:19.519776 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:19.519780 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:19.519922 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:19.520321 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:19.520326 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:19.520329 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:19.520586 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:19.520712 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:19.521391 (+ 679us) server_negotiation.cc:299] Negotiation successful
0620 18:38:19.521603 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 18:38:19.522895 26908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:19.504407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:40761 (local address 127.23.115.3:35083)
0620 18:38:19.505226 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:19.505271 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:19.507110 (+ 1839us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:19.507775 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:19.507784 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:19.508315 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:19.508857 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:19.508873 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:19.510147 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:19.510151 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:19.511647 (+ 1496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:19.511660 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:19.512591 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:19.512598 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:19.512830 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:19.513411 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:19.513433 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:19.515622 (+ 2189us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:19.519197 (+ 3575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:19.519206 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:19.519219 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:19.519650 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:19.520022 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:19.520026 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:19.520028 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:19.520209 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:19.520704 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:19.520712 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:19.521037 (+ 325us) client_negotiation.cc:769] Sending connection context
0620 18:38:19.521268 (+ 231us) client_negotiation.cc:240] Negotiation successful
0620 18:38:19.521800 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":357,"threads_started":1}
I20260620 18:38:19.524233 26906 heartbeater.cc:347] Connected to a master server at 127.23.115.62:40761
I20260620 18:38:19.524505 26906 heartbeater.cc:464] Registering TS with master...
I20260620 18:38:19.525168 26906 heartbeater.cc:511] Master 127.23.115.62:40761 requested a full tablet report, sending...
I20260620 18:38:19.526661 26450 ts_manager.cc:194] Registered new tserver with Master: 681a621c11ba4711ad7e61cac1898745 (127.23.115.3:42271)
I20260620 18:38:19.527554 26450 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.115.3@KRBTEST.COM'} at 127.23.115.3:35083
I20260620 18:38:19.531095 24012 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 18:38:19.558195 26641 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:38:19.555987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38996 (local address 127.23.115.62:40761)
0620 18:38:19.556230 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:19.556236 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:19.556505 (+ 269us) server_negotiation.cc:407] Connection header received
0620 18:38:19.556630 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:19.556634 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:19.556699 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:19.556785 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:19.557945 (+ 1160us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38996: BlockingRecv error: recv got EOF from 127.0.0.1:38996 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 18:38:19.561733 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 26511
W20260620 18:38:19.566020 24012 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 26511: Not authorized: Client connection negotiation failed: client connection to 127.23.115.1:40307: server requires authentication, but client does not have Kerberos credentials available
I20260620 18:38:19.566154 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 26511
I20260620 18:38:19.578717 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 26645
W20260620 18:38:19.581312 24012 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 26645: Not authorized: Client connection negotiation failed: client connection to 127.23.115.2:37977: server requires authentication, but client does not have Kerberos credentials available
I20260620 18:38:19.581444 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 26645
I20260620 18:38:19.593197 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 26778
W20260620 18:38:19.596042 24012 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 26778: Not authorized: Client connection negotiation failed: client connection to 127.23.115.3:42271: server requires authentication, but client does not have Kerberos credentials available
I20260620 18:38:19.596184 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 26778
I20260620 18:38:19.608466 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 26418
I20260620 18:38:19.609551 26480 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:38:19.709707 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 26418
2026-06-20T18:38:19Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3334 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980676253068-24012-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:38:19 dist-test-slave-d645 krb5kdc[26923](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:38:19 dist-test-slave-d645 krb5kdc[26923](info): set up 2 sockets
Jun 20 18:38:19 dist-test-slave-d645 krb5kdc[26923](info): commencing operation
krb5kdc: starting...
W20260620 18:38:20.712188 24012 mini_kdc.cc:121] Time spent starting KDC: real 0.965s user 0.001s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980676253068-24012-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980676253068-24012-0/krb5kdc/test-user.keytab.
Jun 20 18:38:20 dist-test-slave-d645 krb5kdc[26923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980700, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:38:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:38:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.115.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.115.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.115.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab.
Entry for principal kudu/127.23.115.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab.
I20260620 18:38:20.892832 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980676253068-24012-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980676253068-24012-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980676253068-24012-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.115.62:45027
--webserver_interface=127.23.115.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab
--principal=kudu/127.23.115.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:41511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.115.62:45027
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:38:21.206429 26936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:21.206709 26936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:21.206841 26936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:21.213958 26936 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:38:21.214061 26936 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:21.214134 26936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:21.214205 26936 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:38:21.214272 26936 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:38:21.223925 26936 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:41511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980676253068-24012-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980676253068-24012-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.115.62:45027
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab
--principal=kudu/127.23.115.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.115.62:45027
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.115.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980676253068-24012-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:38:21.225147 26936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:21.226363 26936 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:21.237566 26941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:21.237529 26942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:21.237591 26944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:21.238818 26936 server_base.cc:1034] running on GCE node
I20260620 18:38:21.239626 26936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:21.241022 26936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:21.242249 26936 hybrid_clock.cc:648] HybridClock initialized: now 1781980701242198 us; error 42 us; skew 500 ppm
I20260620 18:38:21.242599 26936 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:38:21 dist-test-slave-d645 krb5kdc[26923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980701, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:21.250360 26936 init.cc:377] Logged in from keytab as kudu/127.23.115.62@KRBTEST.COM (short username kudu)
I20260620 18:38:21.252521 26936 webserver.cc:460] Webserver started at http://127.23.115.62:38841/ using document root <none> and password file <none>
I20260620 18:38:21.253244 26936 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:21.253371 26936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:21.253676 26936 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:38:21.257393 26936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980676253068-24012-0/minicluster-data/master-0/data/instance:
uuid: "2634c4940ed7426e8751da216fd399d9"
format_stamp: "Formatted at 2026-06-20 18:38:21 on dist-test-slave-d645"
server_key: "4e994cbe34ab868cb66b3ca995d7e924"
server_key_iv: "fe07b5b33cefa9fc08ffdce5907f73e6"
server_key_version: "encryptionkey@0"
I20260620 18:38:21.258563 26936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980676253068-24012-0/minicluster-data/master-0/wal/instance:
uuid: "2634c4940ed7426e8751da216fd399d9"
format_stamp: "Formatted at 2026-06-20 18:38:21 on dist-test-slave-d645"
server_key: "4e994cbe34ab868cb66b3ca995d7e924"
server_key_iv: "fe07b5b33cefa9fc08ffdce5907f73e6"
server_key_version: "encryptionkey@0"
I20260620 18:38:21.263904 26936 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 18:38:21.267321 26951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:21.268646 26936 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 18:38:21.268807 26936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980676253068-24012-0/minicluster-data/master-0/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980676253068-24012-0/minicluster-data/master-0/wal
uuid: "2634c4940ed7426e8751da216fd399d9"
format_stamp: "Formatted at 2026-06-20 18:38:21 on dist-test-slave-d645"
server_key: "4e994cbe34ab868cb66b3ca995d7e924"
server_key_iv: "fe07b5b33cefa9fc08ffdce5907f73e6"
server_key_version: "encryptionkey@0"
I20260620 18:38:21.268990 26936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980676253068-24012-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980676253068-24012-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980676253068-24012-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:21.290277 26936 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:21.294595 26936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:21.294873 26936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:21.312899 26936 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.62:45027
I20260620 18:38:21.312932 27002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.62:45027 every 8 connection(s)
I20260620 18:38:21.314172 26936 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
I20260620 18:38:21.317737 27003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:21.320135 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 26936
I20260620 18:38:21.320353 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980676253068-24012-0/minicluster-data/master-0/wal/instance
I20260620 18:38:21.320961 24012 external_mini_cluster.cc:1428] Setting key 64b366941e81aca69c411683bffdc30e
I20260620 18:38:21.330654 27003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2634c4940ed7426e8751da216fd399d9: Bootstrap starting.
I20260620 18:38:21.334335 27003 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2634c4940ed7426e8751da216fd399d9: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:21.335631 27003 log.cc:824] T 00000000000000000000000000000000 P 2634c4940ed7426e8751da216fd399d9: Log is configured to *not* fsync() on all Append() calls
Jun 20 18:38:21 dist-test-slave-d645 krb5kdc[26923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980700, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.115.62@KRBTEST.COM
I20260620 18:38:21.338529 27003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2634c4940ed7426e8751da216fd399d9: No bootstrap required, opened a new log
I20260620 18:38:21.344377 27006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:21.322689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38060 (local address 127.23.115.62:45027)
0620 18:38:21.323477 (+ 788us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:21.323492 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:21.323539 (+ 47us) server_negotiation.cc:407] Connection header received
0620 18:38:21.325519 (+ 1980us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:21.325545 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:21.325943 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:21.326260 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:21.328674 (+ 2414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:21.330057 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:21.331813 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:21.332521 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:21.332659 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:21.337415 (+ 4756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:21.337424 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:21.337440 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:21.337477 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:21.340341 (+ 2864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:21.341034 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:21.341041 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:21.341047 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:21.341274 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:21.341627 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:21.341630 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:21.341632 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:21.342157 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:21.342470 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:21.342916 (+ 446us) server_negotiation.cc:299] Negotiation successful
0620 18:38:21.343355 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":372,"threads_started":1}
I20260620 18:38:21.346664 27003 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2634c4940ed7426e8751da216fd399d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2634c4940ed7426e8751da216fd399d9" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 45027 } }
I20260620 18:38:21.347045 27003 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2634c4940ed7426e8751da216fd399d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:21.347163 27003 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2634c4940ed7426e8751da216fd399d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2634c4940ed7426e8751da216fd399d9, State: Initialized, Role: FOLLOWER
I20260620 18:38:21.347700 27003 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2634c4940ed7426e8751da216fd399d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2634c4940ed7426e8751da216fd399d9" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 45027 } }
I20260620 18:38:21.347967 27003 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2634c4940ed7426e8751da216fd399d9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:38:21.348112 27003 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2634c4940ed7426e8751da216fd399d9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:38:21.348286 27003 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2634c4940ed7426e8751da216fd399d9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:21.348338 24012 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:38:21.351384 27003 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2634c4940ed7426e8751da216fd399d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2634c4940ed7426e8751da216fd399d9" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 45027 } }
I20260620 18:38:21.351954 27003 leader_election.cc:302] T 00000000000000000000000000000000 P 2634c4940ed7426e8751da216fd399d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2634c4940ed7426e8751da216fd399d9; no voters:
I20260620 18:38:21.352594 27003 leader_election.cc:288] T 00000000000000000000000000000000 P 2634c4940ed7426e8751da216fd399d9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:38:21.352723 27008 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2634c4940ed7426e8751da216fd399d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:38:21.353822 27008 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2634c4940ed7426e8751da216fd399d9 [term 1 LEADER]: Becoming Leader. State: Replica: 2634c4940ed7426e8751da216fd399d9, State: Running, Role: LEADER
I20260620 18:38:21.354323 27008 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2634c4940ed7426e8751da216fd399d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2634c4940ed7426e8751da216fd399d9" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 45027 } }
I20260620 18:38:21.354640 27003 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2634c4940ed7426e8751da216fd399d9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:38:21.358841 27009 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2634c4940ed7426e8751da216fd399d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2634c4940ed7426e8751da216fd399d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2634c4940ed7426e8751da216fd399d9" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 45027 } } }
I20260620 18:38:21.358866 27010 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2634c4940ed7426e8751da216fd399d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2634c4940ed7426e8751da216fd399d9. Latest consensus state: current_term: 1 leader_uuid: "2634c4940ed7426e8751da216fd399d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2634c4940ed7426e8751da216fd399d9" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 45027 } } }
I20260620 18:38:21.359380 27010 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2634c4940ed7426e8751da216fd399d9 [sys.catalog]: This master's current role is: LEADER
I20260620 18:38:21.359280 27009 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2634c4940ed7426e8751da216fd399d9 [sys.catalog]: This master's current role is: LEADER
I20260620 18:38:21.360225 27017 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:38:21.364813 27017 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:38:21.390429 27017 catalog_manager.cc:1347] Generated new cluster ID: dab0607bda1447788a6cdd89cf6988d9
I20260620 18:38:21.390552 27017 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:38:21.400779 27017 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:38:21.402716 27017 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:38:21.412125 27033 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 18:38:21.419756 27006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:21.408060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38074 (local address 127.23.115.62:45027)
0620 18:38:21.408257 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:21.408265 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:21.413387 (+ 5122us) server_negotiation.cc:407] Connection header received
0620 18:38:21.413556 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:21.413561 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:21.413630 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:21.413716 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:21.414713 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:21.415563 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:21.416804 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:21.417209 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:21.417337 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:21.418309 (+ 972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:21.418314 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:21.418317 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:38:21.418979 (+ 662us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:21.419047 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:38:21.419074 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:21.419166 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:21.419301 (+ 135us) server_negotiation.cc:299] Negotiation successful
0620 18:38:21.419403 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":74}
I20260620 18:38:21.423606 27017 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2634c4940ed7426e8751da216fd399d9: Generated new TSK 0
I20260620 18:38:21.424609 27017 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:38:21.432870 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 26936
I20260620 18:38:21.434116 26998 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:38:21.522637 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 26936
2026-06-20T18:38:21Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1807 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980676253068-24012-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:38:21 dist-test-slave-d645 krb5kdc[27038](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:38:21 dist-test-slave-d645 krb5kdc[27038](info): set up 2 sockets
Jun 20 18:38:21 dist-test-slave-d645 krb5kdc[27038](info): commencing operation
krb5kdc: starting...
W20260620 18:38:22.519277 24012 mini_kdc.cc:121] Time spent starting KDC: real 0.965s user 0.005s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980676253068-24012-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980676253068-24012-0/krb5kdc/test-user.keytab.
Jun 20 18:38:22 dist-test-slave-d645 krb5kdc[27038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980702, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:38:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:38:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.115.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.115.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.115.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab.
Entry for principal kudu/127.23.115.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab.
I20260620 18:38:22.698280 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980676253068-24012-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980676253068-24012-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980676253068-24012-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.115.62:40931
--webserver_interface=127.23.115.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab
--principal=kudu/127.23.115.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:34987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.115.62:40931
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:38:23.009413 27051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:23.009884 27051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:23.010222 27051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:23.018069 27051 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:38:23.018195 27051 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:23.018245 27051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:23.018322 27051 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:38:23.018381 27051 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:38:23.028568 27051 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:34987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980676253068-24012-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980676253068-24012-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.115.62:40931
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab
--principal=kudu/127.23.115.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.115.62:40931
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.115.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980676253068-24012-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:38:23.029889 27051 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:23.031145 27051 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:23.042333 27059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:23.042412 27056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:23.042462 27057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:23.043536 27051 server_base.cc:1034] running on GCE node
I20260620 18:38:23.044248 27051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:23.045621 27051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:23.046833 27051 hybrid_clock.cc:648] HybridClock initialized: now 1781980703046777 us; error 33 us; skew 500 ppm
I20260620 18:38:23.047199 27051 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:38:23 dist-test-slave-d645 krb5kdc[27038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980703, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:23.050137 27051 init.cc:377] Logged in from keytab as kudu/127.23.115.62@KRBTEST.COM (short username kudu)
I20260620 18:38:23.051597 27051 webserver.cc:460] Webserver started at http://127.23.115.62:39551/ using document root <none> and password file <none>
I20260620 18:38:23.052229 27051 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:23.052345 27051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:23.052639 27051 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:38:23.056068 27051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980676253068-24012-0/minicluster-data/master-0/data/instance:
uuid: "6ddd315089514c42b15012b0440a9999"
format_stamp: "Formatted at 2026-06-20 18:38:23 on dist-test-slave-d645"
server_key: "9e8f5f489d6c41fbe98aaa5f49e4017b"
server_key_iv: "36cafca741438f71ba1025c47bc786c4"
server_key_version: "encryptionkey@0"
I20260620 18:38:23.056896 27051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980676253068-24012-0/minicluster-data/master-0/wal/instance:
uuid: "6ddd315089514c42b15012b0440a9999"
format_stamp: "Formatted at 2026-06-20 18:38:23 on dist-test-slave-d645"
server_key: "9e8f5f489d6c41fbe98aaa5f49e4017b"
server_key_iv: "36cafca741438f71ba1025c47bc786c4"
server_key_version: "encryptionkey@0"
I20260620 18:38:23.062139 27051 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 18:38:23.065675 27066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:23.067173 27051 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 18:38:23.067323 27051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980676253068-24012-0/minicluster-data/master-0/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980676253068-24012-0/minicluster-data/master-0/wal
uuid: "6ddd315089514c42b15012b0440a9999"
format_stamp: "Formatted at 2026-06-20 18:38:23 on dist-test-slave-d645"
server_key: "9e8f5f489d6c41fbe98aaa5f49e4017b"
server_key_iv: "36cafca741438f71ba1025c47bc786c4"
server_key_version: "encryptionkey@0"
I20260620 18:38:23.067493 27051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980676253068-24012-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980676253068-24012-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980676253068-24012-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:23.095489 27051 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:23.099558 27051 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:23.099803 27051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:23.117375 27051 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.62:40931
I20260620 18:38:23.117395 27117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.62:40931 every 8 connection(s)
I20260620 18:38:23.118664 27051 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
I20260620 18:38:23.122309 27118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:23.127353 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 27051
I20260620 18:38:23.127557 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980676253068-24012-0/minicluster-data/master-0/wal/instance
I20260620 18:38:23.128269 24012 external_mini_cluster.cc:1428] Setting key b4a57562b7466bd1c3a0807563ce2b51
I20260620 18:38:23.133103 27118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ddd315089514c42b15012b0440a9999: Bootstrap starting.
I20260620 18:38:23.136538 27118 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ddd315089514c42b15012b0440a9999: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:23.137789 27118 log.cc:824] T 00000000000000000000000000000000 P 6ddd315089514c42b15012b0440a9999: Log is configured to *not* fsync() on all Append() calls
I20260620 18:38:23.141198 27118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ddd315089514c42b15012b0440a9999: No bootstrap required, opened a new log
Jun 20 18:38:23 dist-test-slave-d645 krb5kdc[27038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980702, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.115.62@KRBTEST.COM
I20260620 18:38:23.150466 27118 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ddd315089514c42b15012b0440a9999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ddd315089514c42b15012b0440a9999" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 40931 } }
I20260620 18:38:23.151026 27118 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6ddd315089514c42b15012b0440a9999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:23.151136 27118 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ddd315089514c42b15012b0440a9999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ddd315089514c42b15012b0440a9999, State: Initialized, Role: FOLLOWER
I20260620 18:38:23.151567 27121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:23.130264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32826 (local address 127.23.115.62:40931)
0620 18:38:23.131062 (+ 798us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:23.131073 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:23.131116 (+ 43us) server_negotiation.cc:407] Connection header received
0620 18:38:23.132697 (+ 1581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:23.132717 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:23.133081 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:23.133395 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:23.135079 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:23.136371 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:23.138033 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:23.139040 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:23.139220 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:23.143929 (+ 4709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:23.143937 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:23.143953 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:23.143995 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:23.147378 (+ 3383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:23.148208 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:23.148215 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:23.148225 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:23.148427 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:23.148828 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:23.148831 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:23.148833 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:23.149415 (+ 582us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:23.149641 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:23.150068 (+ 427us) server_negotiation.cc:299] Negotiation successful
0620 18:38:23.150437 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":344,"threads_started":1}
I20260620 18:38:23.151801 27118 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ddd315089514c42b15012b0440a9999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ddd315089514c42b15012b0440a9999" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 40931 } }
I20260620 18:38:23.152040 27118 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6ddd315089514c42b15012b0440a9999 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:38:23.152171 27118 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6ddd315089514c42b15012b0440a9999 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:38:23.152334 27118 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ddd315089514c42b15012b0440a9999 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:23.155323 27118 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6ddd315089514c42b15012b0440a9999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ddd315089514c42b15012b0440a9999" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 40931 } }
I20260620 18:38:23.155856 27118 leader_election.cc:302] T 00000000000000000000000000000000 P 6ddd315089514c42b15012b0440a9999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6ddd315089514c42b15012b0440a9999; no voters:
I20260620 18:38:23.156402 27118 leader_election.cc:288] T 00000000000000000000000000000000 P 6ddd315089514c42b15012b0440a9999 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:38:23.156530 27123 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6ddd315089514c42b15012b0440a9999 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:38:23.157157 27123 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6ddd315089514c42b15012b0440a9999 [term 1 LEADER]: Becoming Leader. State: Replica: 6ddd315089514c42b15012b0440a9999, State: Running, Role: LEADER
I20260620 18:38:23.157666 27123 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ddd315089514c42b15012b0440a9999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ddd315089514c42b15012b0440a9999" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 40931 } }
I20260620 18:38:23.158352 27118 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ddd315089514c42b15012b0440a9999 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:38:23.161077 27125 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ddd315089514c42b15012b0440a9999 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ddd315089514c42b15012b0440a9999. Latest consensus state: current_term: 1 leader_uuid: "6ddd315089514c42b15012b0440a9999" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ddd315089514c42b15012b0440a9999" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 40931 } } }
I20260620 18:38:23.161571 27125 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ddd315089514c42b15012b0440a9999 [sys.catalog]: This master's current role is: LEADER
I20260620 18:38:23.162309 27124 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ddd315089514c42b15012b0440a9999 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6ddd315089514c42b15012b0440a9999" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ddd315089514c42b15012b0440a9999" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 40931 } } }
I20260620 18:38:23.163134 27124 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ddd315089514c42b15012b0440a9999 [sys.catalog]: This master's current role is: LEADER
I20260620 18:38:23.166226 27128 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:38:23.172106 27128 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:38:23.180986 27128 catalog_manager.cc:1347] Generated new cluster ID: 4904456910874670a3b843ab3802e757
I20260620 18:38:23.181157 27128 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:38:23.200318 27128 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:38:23.202214 27128 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:38:23.211191 27128 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6ddd315089514c42b15012b0440a9999: Generated new TSK 0
I20260620 18:38:23.212245 27128 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.115.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.115.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.115.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.1.keytab.
Entry for principal kudu/127.23.115.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.1.keytab.
I20260620 18:38:23.258886 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980676253068-24012-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980676253068-24012-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980676253068-24012-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980676253068-24012-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.115.1:0
--local_ip_for_outbound_sockets=127.23.115.1
--webserver_interface=127.23.115.1
--webserver_port=0
--tserver_master_addrs=127.23.115.62:40931
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.1.keytab
--principal=kudu/127.23.115.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:34987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:38:23.579219 27144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:23.579519 27144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:23.579638 27144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:23.586759 27144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:23.586862 27144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:23.587020 27144 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.115.1
I20260620 18:38:23.596890 27144 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:34987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980676253068-24012-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980676253068-24012-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.1.keytab
--principal=kudu/127.23.115.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.115.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980676253068-24012-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.115.1
--webserver_port=0
--tserver_master_addrs=127.23.115.62:40931
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.115.1
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980676253068-24012-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:38:23.598191 27144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:23.599499 27144 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:23.613178 27149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:23.613085 27152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:23.613135 27150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:23.613629 27144 server_base.cc:1034] running on GCE node
I20260620 18:38:23.614289 27144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:23.615275 27144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:23.616513 27144 hybrid_clock.cc:648] HybridClock initialized: now 1781980703616466 us; error 40 us; skew 500 ppm
I20260620 18:38:23.616858 27144 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:38:23 dist-test-slave-d645 krb5kdc[27038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980703, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:23.620064 27144 init.cc:377] Logged in from keytab as kudu/127.23.115.1@KRBTEST.COM (short username kudu)
I20260620 18:38:23.621636 27144 webserver.cc:460] Webserver started at http://127.23.115.1:34485/ using document root <none> and password file <none>
I20260620 18:38:23.622292 27144 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:23.622406 27144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:23.622696 27144 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:38:23.625593 27144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980676253068-24012-0/minicluster-data/ts-0/data/instance:
uuid: "5ee5b9fe2a274e66a62d529bcdada4ed"
format_stamp: "Formatted at 2026-06-20 18:38:23 on dist-test-slave-d645"
server_key: "add96ca62b09607eac553ca0701c944e"
server_key_iv: "0453474634f1961892afd3e3800b4c8c"
server_key_version: "encryptionkey@0"
I20260620 18:38:23.626343 27144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980676253068-24012-0/minicluster-data/ts-0/wal/instance:
uuid: "5ee5b9fe2a274e66a62d529bcdada4ed"
format_stamp: "Formatted at 2026-06-20 18:38:23 on dist-test-slave-d645"
server_key: "add96ca62b09607eac553ca0701c944e"
server_key_iv: "0453474634f1961892afd3e3800b4c8c"
server_key_version: "encryptionkey@0"
I20260620 18:38:23.631350 27144 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.006s
I20260620 18:38:23.634629 27159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:23.635908 27144 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:38:23.636051 27144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980676253068-24012-0/minicluster-data/ts-0/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980676253068-24012-0/minicluster-data/ts-0/wal
uuid: "5ee5b9fe2a274e66a62d529bcdada4ed"
format_stamp: "Formatted at 2026-06-20 18:38:23 on dist-test-slave-d645"
server_key: "add96ca62b09607eac553ca0701c944e"
server_key_iv: "0453474634f1961892afd3e3800b4c8c"
server_key_version: "encryptionkey@0"
I20260620 18:38:23.636219 27144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980676253068-24012-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980676253068-24012-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980676253068-24012-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:23.655514 27144 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:23.659507 27144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:23.659780 27144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:23.660702 27144 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:38:23.661979 27144 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:38:23.662053 27144 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:23.662173 27144 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:38:23.662223 27144 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:23.692678 27144 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.1:44687
I20260620 18:38:23.692855 27271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.1:44687 every 8 connection(s)
I20260620 18:38:23.693934 27144 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980676253068-24012-0/minicluster-data/ts-0/data/info.pb
I20260620 18:38:23.698091 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 27144
I20260620 18:38:23.698263 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980676253068-24012-0/minicluster-data/ts-0/wal/instance
I20260620 18:38:23.698858 24012 external_mini_cluster.cc:1428] Setting key 87f3468c01234a54867f168a5a36be64
Jun 20 18:38:23 dist-test-slave-d645 krb5kdc[27038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980703, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.1@KRBTEST.COM for kudu/127.23.115.62@KRBTEST.COM
I20260620 18:38:23.714973 27274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:23.696330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.1:36861 (local address 127.23.115.62:40931)
0620 18:38:23.696743 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:23.696748 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:23.697601 (+ 853us) server_negotiation.cc:407] Connection header received
0620 18:38:23.699532 (+ 1931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:23.699537 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:23.699616 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:23.699728 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:23.701446 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:23.702486 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:23.704050 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:23.704608 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:23.704721 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:23.708822 (+ 4101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:23.708827 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:23.708831 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:23.708863 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:23.711598 (+ 2735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:23.712521 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:23.712525 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:23.712528 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:23.712658 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:23.713142 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:23.713145 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:23.713147 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:23.713428 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:23.713562 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:23.714357 (+ 795us) server_negotiation.cc:299] Negotiation successful
0620 18:38:23.714592 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":203,"threads_started":1}
WARNING: no policy specified for kudu/127.23.115.2@KRBTEST.COM; defaulting to no policy
I20260620 18:38:23.715941 27275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:23.696795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:40931 (local address 127.23.115.1:36861)
0620 18:38:23.697451 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:23.697487 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:23.699261 (+ 1774us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:23.699967 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:23.699978 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:23.700689 (+ 711us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:23.701278 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:23.701296 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:23.702646 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:23.702650 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:23.703921 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:23.703933 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:23.704826 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:23.704832 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:23.705064 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:23.705737 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:23.705796 (+ 59us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:23.708528 (+ 2732us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:23.711738 (+ 3210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:23.711747 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:23.711764 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:23.712368 (+ 604us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:23.712769 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:23.712774 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:23.712777 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:23.713038 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:23.713611 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:23.713620 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:23.714002 (+ 382us) client_negotiation.cc:769] Sending connection context
0620 18:38:23.714240 (+ 238us) client_negotiation.cc:240] Negotiation successful
0620 18:38:23.714781 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":247,"threads_started":1}
I20260620 18:38:23.717468 27272 heartbeater.cc:347] Connected to a master server at 127.23.115.62:40931
I20260620 18:38:23.717742 27272 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.115.2@KRBTEST.COM" created.
I20260620 18:38:23.718461 27272 heartbeater.cc:511] Master 127.23.115.62:40931 requested a full tablet report, sending...
I20260620 18:38:23.720587 27083 ts_manager.cc:194] Registered new tserver with Master: 5ee5b9fe2a274e66a62d529bcdada4ed (127.23.115.1:44687)
I20260620 18:38:23.722342 27083 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.115.1@KRBTEST.COM'} at 127.23.115.1:36861
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.115.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.2.keytab.
Entry for principal kudu/127.23.115.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.2.keytab.
I20260620 18:38:23.740170 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980676253068-24012-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980676253068-24012-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980676253068-24012-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980676253068-24012-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.115.2:0
--local_ip_for_outbound_sockets=127.23.115.2
--webserver_interface=127.23.115.2
--webserver_port=0
--tserver_master_addrs=127.23.115.62:40931
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.2.keytab
--principal=kudu/127.23.115.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:34987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:38:24.045718 27278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:24.045979 27278 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:24.046097 27278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:24.053182 27278 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:24.053285 27278 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:24.053440 27278 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.115.2
I20260620 18:38:24.063094 27278 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:34987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980676253068-24012-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980676253068-24012-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.2.keytab
--principal=kudu/127.23.115.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.115.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980676253068-24012-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.115.2
--webserver_port=0
--tserver_master_addrs=127.23.115.62:40931
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.115.2
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980676253068-24012-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:38:24.064333 27278 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:24.065546 27278 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:24.078850 27283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:24.078783 27284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:24.078823 27286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:24.079702 27278 server_base.cc:1034] running on GCE node
I20260620 18:38:24.080245 27278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:24.081127 27278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:24.082340 27278 hybrid_clock.cc:648] HybridClock initialized: now 1781980704082272 us; error 53 us; skew 500 ppm
I20260620 18:38:24.082657 27278 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:38:24 dist-test-slave-d645 krb5kdc[27038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980704, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:24.086097 27278 init.cc:377] Logged in from keytab as kudu/127.23.115.2@KRBTEST.COM (short username kudu)
I20260620 18:38:24.087656 27278 webserver.cc:460] Webserver started at http://127.23.115.2:33015/ using document root <none> and password file <none>
I20260620 18:38:24.088305 27278 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:24.088428 27278 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:24.088708 27278 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:38:24.091527 27278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980676253068-24012-0/minicluster-data/ts-1/data/instance:
uuid: "a005a57f54bd4dad8c2c8d1bf8ad2def"
format_stamp: "Formatted at 2026-06-20 18:38:24 on dist-test-slave-d645"
server_key: "f8a54ad93d9ea2a2527cb379442c2fd0"
server_key_iv: "69973cff97ff15f4363d3008ad9f3c32"
server_key_version: "encryptionkey@0"
I20260620 18:38:24.092249 27278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980676253068-24012-0/minicluster-data/ts-1/wal/instance:
uuid: "a005a57f54bd4dad8c2c8d1bf8ad2def"
format_stamp: "Formatted at 2026-06-20 18:38:24 on dist-test-slave-d645"
server_key: "f8a54ad93d9ea2a2527cb379442c2fd0"
server_key_iv: "69973cff97ff15f4363d3008ad9f3c32"
server_key_version: "encryptionkey@0"
I20260620 18:38:24.097175 27278 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260620 18:38:24.100528 27293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:24.101747 27278 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 18:38:24.101912 27278 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980676253068-24012-0/minicluster-data/ts-1/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980676253068-24012-0/minicluster-data/ts-1/wal
uuid: "a005a57f54bd4dad8c2c8d1bf8ad2def"
format_stamp: "Formatted at 2026-06-20 18:38:24 on dist-test-slave-d645"
server_key: "f8a54ad93d9ea2a2527cb379442c2fd0"
server_key_iv: "69973cff97ff15f4363d3008ad9f3c32"
server_key_version: "encryptionkey@0"
I20260620 18:38:24.102082 27278 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980676253068-24012-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980676253068-24012-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980676253068-24012-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:24.116242 27278 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:24.120329 27278 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:24.120582 27278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:24.121495 27278 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:38:24.122795 27278 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:38:24.122879 27278 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:24.123013 27278 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:38:24.123068 27278 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:24.152980 27278 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.2:46369
I20260620 18:38:24.153080 27405 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.2:46369 every 8 connection(s)
I20260620 18:38:24.154173 27278 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980676253068-24012-0/minicluster-data/ts-1/data/info.pb
I20260620 18:38:24.157116 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 27278
I20260620 18:38:24.157312 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980676253068-24012-0/minicluster-data/ts-1/wal/instance
I20260620 18:38:24.157842 24012 external_mini_cluster.cc:1428] Setting key d28f60f317b48888785699536e0605fa
Jun 20 18:38:24 dist-test-slave-d645 krb5kdc[27038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980704, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.2@KRBTEST.COM for kudu/127.23.115.62@KRBTEST.COM
I20260620 18:38:24.173695 27274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:24.156510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.2:43773 (local address 127.23.115.62:40931)
0620 18:38:24.156675 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:24.156679 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:24.157852 (+ 1173us) server_negotiation.cc:407] Connection header received
0620 18:38:24.159857 (+ 2005us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:24.159862 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:24.159949 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:24.160030 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:24.161423 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:24.162131 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:24.163663 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:24.164065 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:24.164169 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:24.167979 (+ 3810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:24.167984 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:24.167987 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:24.168017 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:24.170499 (+ 2482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:24.171393 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:24.171398 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:24.171401 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:24.171526 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:24.171997 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:24.172001 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:24.172004 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:24.172311 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:24.172426 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:24.173235 (+ 809us) server_negotiation.cc:299] Negotiation successful
0620 18:38:24.173438 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
WARNING: no policy specified for kudu/127.23.115.3@KRBTEST.COM; defaulting to no policy
I20260620 18:38:24.174922 27408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:24.156876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:40931 (local address 127.23.115.2:43773)
0620 18:38:24.157698 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:24.157739 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:24.159630 (+ 1891us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:24.160255 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:24.160263 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:24.160739 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:24.161262 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:24.161278 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:24.162279 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:24.162283 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:24.163533 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:24.163545 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:24.164266 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:24.164274 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:24.164493 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:24.165083 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:24.165110 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:24.167747 (+ 2637us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:24.170637 (+ 2890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:24.170648 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:24.170669 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:24.171262 (+ 593us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:24.171627 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:24.171632 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:24.171635 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:24.171867 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:24.172445 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:24.172454 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:24.172867 (+ 413us) client_negotiation.cc:769] Sending connection context
0620 18:38:24.173156 (+ 289us) client_negotiation.cc:240] Negotiation successful
0620 18:38:24.173717 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":351,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.115.3@KRBTEST.COM" created.
I20260620 18:38:24.176390 27406 heartbeater.cc:347] Connected to a master server at 127.23.115.62:40931
I20260620 18:38:24.176689 27406 heartbeater.cc:464] Registering TS with master...
I20260620 18:38:24.177390 27406 heartbeater.cc:511] Master 127.23.115.62:40931 requested a full tablet report, sending...
I20260620 18:38:24.178803 27083 ts_manager.cc:194] Registered new tserver with Master: a005a57f54bd4dad8c2c8d1bf8ad2def (127.23.115.2:46369)
I20260620 18:38:24.179840 27083 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.115.2@KRBTEST.COM'} at 127.23.115.2:43773
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.115.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.3.keytab.
Entry for principal kudu/127.23.115.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.3.keytab.
I20260620 18:38:24.197858 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980676253068-24012-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980676253068-24012-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980676253068-24012-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980676253068-24012-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.115.3:0
--local_ip_for_outbound_sockets=127.23.115.3
--webserver_interface=127.23.115.3
--webserver_port=0
--tserver_master_addrs=127.23.115.62:40931
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.3.keytab
--principal=kudu/127.23.115.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:34987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:38:24.519081 27411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:24.519354 27411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:24.519487 27411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:24.526548 27411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:24.526646 27411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:24.526789 27411 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.115.3
I20260620 18:38:24.536293 27411 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:34987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980676253068-24012-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980676253068-24012-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.3.keytab
--principal=kudu/127.23.115.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.115.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980676253068-24012-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.115.3
--webserver_port=0
--tserver_master_addrs=127.23.115.62:40931
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.115.3
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980676253068-24012-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:38:24.537603 27411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:24.538813 27411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:24.550968 27416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:24.550894 27419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:24.550923 27417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:24.551853 27411 server_base.cc:1034] running on GCE node
I20260620 18:38:24.552400 27411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:24.553279 27411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:24.554488 27411 hybrid_clock.cc:648] HybridClock initialized: now 1781980704554442 us; error 44 us; skew 500 ppm
I20260620 18:38:24.554773 27411 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:38:24 dist-test-slave-d645 krb5kdc[27038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980704, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:24.558065 27411 init.cc:377] Logged in from keytab as kudu/127.23.115.3@KRBTEST.COM (short username kudu)
I20260620 18:38:24.559490 27411 webserver.cc:460] Webserver started at http://127.23.115.3:43485/ using document root <none> and password file <none>
I20260620 18:38:24.560175 27411 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:24.560273 27411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:24.560551 27411 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:38:24.563427 27411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980676253068-24012-0/minicluster-data/ts-2/data/instance:
uuid: "ea8710d8fa3341b192c108dbe6dad5fa"
format_stamp: "Formatted at 2026-06-20 18:38:24 on dist-test-slave-d645"
server_key: "4104bec1c272af4bb211e2f50c2cfc8c"
server_key_iv: "e302ca0d482886985c14ed03bee3f9ac"
server_key_version: "encryptionkey@0"
I20260620 18:38:24.564194 27411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980676253068-24012-0/minicluster-data/ts-2/wal/instance:
uuid: "ea8710d8fa3341b192c108dbe6dad5fa"
format_stamp: "Formatted at 2026-06-20 18:38:24 on dist-test-slave-d645"
server_key: "4104bec1c272af4bb211e2f50c2cfc8c"
server_key_iv: "e302ca0d482886985c14ed03bee3f9ac"
server_key_version: "encryptionkey@0"
I20260620 18:38:24.569327 27411 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:38:24.573118 27426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:24.574502 27411 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:38:24.574661 27411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980676253068-24012-0/minicluster-data/ts-2/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980676253068-24012-0/minicluster-data/ts-2/wal
uuid: "ea8710d8fa3341b192c108dbe6dad5fa"
format_stamp: "Formatted at 2026-06-20 18:38:24 on dist-test-slave-d645"
server_key: "4104bec1c272af4bb211e2f50c2cfc8c"
server_key_iv: "e302ca0d482886985c14ed03bee3f9ac"
server_key_version: "encryptionkey@0"
I20260620 18:38:24.574839 27411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980676253068-24012-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980676253068-24012-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980676253068-24012-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:24.609750 27411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:24.613978 27411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:24.614374 27411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:24.615381 27411 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:38:24.616787 27411 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:38:24.616873 27411 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:24.616994 27411 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:38:24.617045 27411 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:24.649030 27411 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.3:41747
I20260620 18:38:24.649083 27538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.3:41747 every 8 connection(s)
I20260620 18:38:24.650260 27411 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980676253068-24012-0/minicluster-data/ts-2/data/info.pb
I20260620 18:38:24.655050 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 27411
I20260620 18:38:24.655223 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980676253068-24012-0/minicluster-data/ts-2/wal/instance
I20260620 18:38:24.655751 24012 external_mini_cluster.cc:1428] Setting key 6b2e94ebe8588561983bc8df2606d6a6
Jun 20 18:38:24 dist-test-slave-d645 krb5kdc[27038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980704, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.3@KRBTEST.COM for kudu/127.23.115.62@KRBTEST.COM
I20260620 18:38:24.670351 27274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:24.652766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.3:40099 (local address 127.23.115.62:40931)
0620 18:38:24.652952 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:24.652957 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:24.654006 (+ 1049us) server_negotiation.cc:407] Connection header received
0620 18:38:24.656347 (+ 2341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:24.656354 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:24.656446 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:24.656550 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:24.658355 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:24.659375 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:24.660980 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:24.661382 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:24.661475 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:24.664772 (+ 3297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:24.664777 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:24.664781 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:24.664812 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:24.667244 (+ 2432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:24.667982 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:24.667990 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:24.667993 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:24.668110 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:24.668590 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:24.668594 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:24.668596 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:24.668898 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:24.669027 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:24.669877 (+ 850us) server_negotiation.cc:299] Negotiation successful
0620 18:38:24.670078 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 18:38:24.671459 27541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:24.653176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:40931 (local address 127.23.115.3:40099)
0620 18:38:24.653850 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:24.653890 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:24.656003 (+ 2113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:24.656798 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:24.656806 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:24.657489 (+ 683us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:24.658195 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:24.658214 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:24.659571 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:24.659575 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:24.660843 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:24.660856 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:24.661597 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:24.661603 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:24.661798 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:24.662404 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:24.662433 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:24.664566 (+ 2133us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:24.667370 (+ 2804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:24.667377 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:24.667392 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:24.667863 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:24.668237 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:24.668241 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:24.668243 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:24.668463 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:24.669037 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:24.669045 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:24.669452 (+ 407us) client_negotiation.cc:769] Sending connection context
0620 18:38:24.669767 (+ 315us) client_negotiation.cc:240] Negotiation successful
0620 18:38:24.670267 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":273,"threads_started":1}
I20260620 18:38:24.672626 27539 heartbeater.cc:347] Connected to a master server at 127.23.115.62:40931
I20260620 18:38:24.672889 27539 heartbeater.cc:464] Registering TS with master...
I20260620 18:38:24.673585 27539 heartbeater.cc:511] Master 127.23.115.62:40931 requested a full tablet report, sending...
I20260620 18:38:24.675108 27083 ts_manager.cc:194] Registered new tserver with Master: ea8710d8fa3341b192c108dbe6dad5fa (127.23.115.3:41747)
I20260620 18:38:24.675983 27083 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.115.3@KRBTEST.COM'} at 127.23.115.3:40099
I20260620 18:38:24.683331 24012 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 18:38:24 dist-test-slave-d645 krb5kdc[27038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980704, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 20 18:38:24 dist-test-slave-d645 krb5kdc[27038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980704, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.23.115.62@KRBTEST.COM
I20260620 18:38:24.725675 27272 heartbeater.cc:503] Master 127.23.115.62:40931 was elected leader, sending a full tablet report...
I20260620 18:38:24.729285 27274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:24.715759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32834 (local address 127.23.115.62:40931)
0620 18:38:24.715952 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:24.715957 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:24.716499 (+ 542us) server_negotiation.cc:407] Connection header received
0620 18:38:24.716646 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:24.716650 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:24.716719 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:24.716800 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:24.718054 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:24.718819 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:24.720134 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:24.720530 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:24.720627 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:24.724314 (+ 3687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:24.724319 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:24.724323 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:24.724354 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:24.726882 (+ 2528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:24.727507 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:24.727512 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:24.727514 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:24.727637 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:24.727970 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:24.727975 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:24.727978 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:24.728263 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:24.728472 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:24.728791 (+ 319us) server_negotiation.cc:299] Negotiation successful
0620 18:38:24.728992 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260620 18:38:24.729796 27083 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:32834
I20260620 18:38:24.731300 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 27144
Jun 20 18:38:24 dist-test-slave-d645 krb5kdc[27038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980704, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.23.115.1@KRBTEST.COM
I20260620 18:38:24.747057 27550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:24.732503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42912 (local address 127.23.115.1:44687)
0620 18:38:24.733090 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:24.733097 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:24.733120 (+ 23us) server_negotiation.cc:407] Connection header received
0620 18:38:24.733175 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:24.733181 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:24.733369 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:24.733489 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:24.734568 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:24.735673 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:24.736859 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:24.737391 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:24.737488 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:24.741586 (+ 4098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:24.741597 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:24.741611 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:24.741645 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:24.744350 (+ 2705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:24.745029 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:24.745036 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:24.745038 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:24.745173 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:24.745553 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:24.745557 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:24.745559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:24.745887 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:24.746155 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:24.746424 (+ 269us) server_negotiation.cc:299] Negotiation successful
0620 18:38:24.746666 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":297,"threads_started":1}
W20260620 18:38:24.748202 27267 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:42912
W20260620 18:38:24.749259 24012 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 27144: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:38:24.749442 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 27144
I20260620 18:38:24.762274 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 27278
Jun 20 18:38:24 dist-test-slave-d645 krb5kdc[27038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980704, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.23.115.2@KRBTEST.COM
I20260620 18:38:24.778321 27551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:24.763139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42014 (local address 127.23.115.2:46369)
0620 18:38:24.763702 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:24.763708 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:24.763733 (+ 25us) server_negotiation.cc:407] Connection header received
0620 18:38:24.763790 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:24.763796 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:24.763999 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:24.764123 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:24.765291 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:24.766339 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:24.768071 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:24.768695 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:24.768811 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:24.772911 (+ 4100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:24.772922 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:24.772936 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:24.772971 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:24.775691 (+ 2720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:24.776299 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:24.776306 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:24.776308 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:24.776421 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:24.776758 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:24.776762 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:24.776764 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:24.777087 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:24.777361 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:24.777682 (+ 321us) server_negotiation.cc:299] Negotiation successful
0620 18:38:24.777935 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":303,"threads_started":1}
W20260620 18:38:24.779560 27401 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:42014
W20260620 18:38:24.780557 24012 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 27278: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:38:24.780681 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 27278
I20260620 18:38:24.792500 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 27411
Jun 20 18:38:24 dist-test-slave-d645 krb5kdc[27038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980704, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.23.115.3@KRBTEST.COM
I20260620 18:38:24.808509 27552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:24.793463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33448 (local address 127.23.115.3:41747)
0620 18:38:24.794035 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:24.794042 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:24.794077 (+ 35us) server_negotiation.cc:407] Connection header received
0620 18:38:24.794139 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:24.794145 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:24.794351 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:24.794475 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:24.795756 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:24.796880 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:24.798104 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:24.798666 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:24.798763 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:24.802974 (+ 4211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:24.802990 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:24.803005 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:24.803041 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:24.805720 (+ 2679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:24.806315 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:24.806322 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:24.806325 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:24.806439 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:24.806835 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:24.806839 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:24.806841 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:24.807253 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:24.807484 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:24.807954 (+ 470us) server_negotiation.cc:299] Negotiation successful
0620 18:38:24.808147 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":273,"threads_started":1}
W20260620 18:38:24.809798 27534 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:33448
W20260620 18:38:24.810854 24012 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 27411: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:38:24.811060 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 27411
I20260620 18:38:24.822962 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 27051
I20260620 18:38:24.824067 27113 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:38:24.926095 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 27051
2026-06-20T18:38:24Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3408 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980676253068-24012-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:38:25 dist-test-slave-d645 krb5kdc[27556](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:38:25 dist-test-slave-d645 krb5kdc[27556](info): set up 2 sockets
Jun 20 18:38:25 dist-test-slave-d645 krb5kdc[27556](info): commencing operation
krb5kdc: starting...
W20260620 18:38:25.929718 24012 mini_kdc.cc:121] Time spent starting KDC: real 0.967s user 0.000s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980676253068-24012-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980676253068-24012-0/krb5kdc/test-user.keytab.
Jun 20 18:38:26 dist-test-slave-d645 krb5kdc[27556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980706, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:38:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:38:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.115.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.115.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.115.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab.
Entry for principal kudu/127.23.115.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab.
I20260620 18:38:26.126981 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980676253068-24012-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980676253068-24012-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980676253068-24012-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.115.62:38609
--webserver_interface=127.23.115.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab
--principal=kudu/127.23.115.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:37099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.115.62:38609
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:38:26.438840 27569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:26.439183 27569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:26.439313 27569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:26.446496 27569 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:38:26.446601 27569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:26.446656 27569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:26.446730 27569 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:38:26.446781 27569 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:38:26.456566 27569 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:37099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980676253068-24012-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980676253068-24012-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.115.62:38609
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab
--principal=kudu/127.23.115.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.115.62:38609
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.115.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980676253068-24012-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:38:26.457864 27569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:26.459419 27569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:26.471009 27574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:26.471058 27575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:26.471171 27577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:26.471866 27569 server_base.cc:1034] running on GCE node
I20260620 18:38:26.472587 27569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:26.473968 27569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:26.475195 27569 hybrid_clock.cc:648] HybridClock initialized: now 1781980706475149 us; error 34 us; skew 500 ppm
I20260620 18:38:26.475564 27569 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:38:26 dist-test-slave-d645 krb5kdc[27556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980706, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:26.483101 27569 init.cc:377] Logged in from keytab as kudu/127.23.115.62@KRBTEST.COM (short username kudu)
I20260620 18:38:26.484613 27569 webserver.cc:460] Webserver started at http://127.23.115.62:44949/ using document root <none> and password file <none>
I20260620 18:38:26.485296 27569 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:26.485410 27569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:26.485718 27569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:38:26.488734 27569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980676253068-24012-0/minicluster-data/master-0/data/instance:
uuid: "6190b90e6aa744c5b01ff34dda13bfc8"
format_stamp: "Formatted at 2026-06-20 18:38:26 on dist-test-slave-d645"
server_key: "fece909adaec01590f3526460f3349c5"
server_key_iv: "8a877ecdfdf94eb20be9f0208ff4162c"
server_key_version: "encryptionkey@0"
I20260620 18:38:26.489573 27569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980676253068-24012-0/minicluster-data/master-0/wal/instance:
uuid: "6190b90e6aa744c5b01ff34dda13bfc8"
format_stamp: "Formatted at 2026-06-20 18:38:26 on dist-test-slave-d645"
server_key: "fece909adaec01590f3526460f3349c5"
server_key_iv: "8a877ecdfdf94eb20be9f0208ff4162c"
server_key_version: "encryptionkey@0"
I20260620 18:38:26.494684 27569 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 18:38:26.498350 27584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:26.499747 27569 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:38:26.499925 27569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980676253068-24012-0/minicluster-data/master-0/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980676253068-24012-0/minicluster-data/master-0/wal
uuid: "6190b90e6aa744c5b01ff34dda13bfc8"
format_stamp: "Formatted at 2026-06-20 18:38:26 on dist-test-slave-d645"
server_key: "fece909adaec01590f3526460f3349c5"
server_key_iv: "8a877ecdfdf94eb20be9f0208ff4162c"
server_key_version: "encryptionkey@0"
I20260620 18:38:26.500125 27569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980676253068-24012-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980676253068-24012-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980676253068-24012-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:26.532397 27569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:26.537319 27569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:26.537786 27569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:26.556844 27569 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.62:38609
I20260620 18:38:26.556883 27635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.62:38609 every 8 connection(s)
I20260620 18:38:26.558118 27569 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
I20260620 18:38:26.561751 27636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:26.565908 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 27569
I20260620 18:38:26.566123 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980676253068-24012-0/minicluster-data/master-0/wal/instance
I20260620 18:38:26.566700 24012 external_mini_cluster.cc:1428] Setting key d4e4bab0f0c62b73251f0c6c251963ef
I20260620 18:38:26.572170 27636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6190b90e6aa744c5b01ff34dda13bfc8: Bootstrap starting.
I20260620 18:38:26.575688 27636 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6190b90e6aa744c5b01ff34dda13bfc8: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:26.576922 27636 log.cc:824] T 00000000000000000000000000000000 P 6190b90e6aa744c5b01ff34dda13bfc8: Log is configured to *not* fsync() on all Append() calls
I20260620 18:38:26.580498 27636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6190b90e6aa744c5b01ff34dda13bfc8: No bootstrap required, opened a new log
Jun 20 18:38:26 dist-test-slave-d645 krb5kdc[27556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980706, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.115.62@KRBTEST.COM
I20260620 18:38:26.588496 27636 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6190b90e6aa744c5b01ff34dda13bfc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6190b90e6aa744c5b01ff34dda13bfc8" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 38609 } }
I20260620 18:38:26.588888 27636 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6190b90e6aa744c5b01ff34dda13bfc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:26.589025 27636 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6190b90e6aa744c5b01ff34dda13bfc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6190b90e6aa744c5b01ff34dda13bfc8, State: Initialized, Role: FOLLOWER
I20260620 18:38:26.589633 27636 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6190b90e6aa744c5b01ff34dda13bfc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6190b90e6aa744c5b01ff34dda13bfc8" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 38609 } }
I20260620 18:38:26.589870 27636 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6190b90e6aa744c5b01ff34dda13bfc8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:38:26.590001 27636 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6190b90e6aa744c5b01ff34dda13bfc8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:38:26.590163 27636 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6190b90e6aa744c5b01ff34dda13bfc8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:26.590207 27639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:26.568816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60162 (local address 127.23.115.62:38609)
0620 18:38:26.569665 (+ 849us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:26.569676 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:26.569721 (+ 45us) server_negotiation.cc:407] Connection header received
0620 18:38:26.571443 (+ 1722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:26.571465 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:26.571840 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:26.572144 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:26.573951 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:26.575400 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:26.577103 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:26.578078 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:26.578274 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:26.582811 (+ 4537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:26.582819 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:26.582868 (+ 49us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:26.582913 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:26.586084 (+ 3171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:26.586808 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:26.586815 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:26.586821 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:26.587051 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:26.587412 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:26.587417 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:26.587420 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:26.587972 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:26.588226 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:26.588668 (+ 442us) server_negotiation.cc:299] Negotiation successful
0620 18:38:26.589121 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":365,"threads_started":1}
I20260620 18:38:26.592934 27636 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6190b90e6aa744c5b01ff34dda13bfc8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6190b90e6aa744c5b01ff34dda13bfc8" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 38609 } }
I20260620 18:38:26.593492 27636 leader_election.cc:302] T 00000000000000000000000000000000 P 6190b90e6aa744c5b01ff34dda13bfc8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6190b90e6aa744c5b01ff34dda13bfc8; no voters:
I20260620 18:38:26.594028 27636 leader_election.cc:288] T 00000000000000000000000000000000 P 6190b90e6aa744c5b01ff34dda13bfc8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:38:26.594250 27641 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6190b90e6aa744c5b01ff34dda13bfc8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:38:26.594911 27641 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6190b90e6aa744c5b01ff34dda13bfc8 [term 1 LEADER]: Becoming Leader. State: Replica: 6190b90e6aa744c5b01ff34dda13bfc8, State: Running, Role: LEADER
I20260620 18:38:26.595445 27641 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6190b90e6aa744c5b01ff34dda13bfc8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6190b90e6aa744c5b01ff34dda13bfc8" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 38609 } }
I20260620 18:38:26.596266 27636 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6190b90e6aa744c5b01ff34dda13bfc8 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:38:26.598577 27642 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6190b90e6aa744c5b01ff34dda13bfc8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6190b90e6aa744c5b01ff34dda13bfc8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6190b90e6aa744c5b01ff34dda13bfc8" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 38609 } } }
I20260620 18:38:26.598778 27643 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6190b90e6aa744c5b01ff34dda13bfc8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6190b90e6aa744c5b01ff34dda13bfc8. Latest consensus state: current_term: 1 leader_uuid: "6190b90e6aa744c5b01ff34dda13bfc8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6190b90e6aa744c5b01ff34dda13bfc8" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 38609 } } }
I20260620 18:38:26.599134 27643 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6190b90e6aa744c5b01ff34dda13bfc8 [sys.catalog]: This master's current role is: LEADER
I20260620 18:38:26.599134 27642 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6190b90e6aa744c5b01ff34dda13bfc8 [sys.catalog]: This master's current role is: LEADER
I20260620 18:38:26.600613 27647 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:38:26.605154 27647 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:38:26.613847 27647 catalog_manager.cc:1347] Generated new cluster ID: 27da4239473149b0a3f5c2165bda86c7
I20260620 18:38:26.614025 27647 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:38:26.632182 27647 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:38:26.633916 27647 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:38:26.643062 27647 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6190b90e6aa744c5b01ff34dda13bfc8: Generated new TSK 0
I20260620 18:38:26.643949 27647 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.115.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.115.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.115.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.1.keytab.
Entry for principal kudu/127.23.115.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.1.keytab.
I20260620 18:38:26.695379 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980676253068-24012-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980676253068-24012-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980676253068-24012-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980676253068-24012-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.115.1:0
--local_ip_for_outbound_sockets=127.23.115.1
--webserver_interface=127.23.115.1
--webserver_port=0
--tserver_master_addrs=127.23.115.62:38609
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.1.keytab
--principal=kudu/127.23.115.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:37099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:38:27.014031 27662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:27.014415 27662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:27.014559 27662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:27.022384 27662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:27.022524 27662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:27.022657 27662 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.115.1
I20260620 18:38:27.033071 27662 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:37099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980676253068-24012-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980676253068-24012-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.1.keytab
--principal=kudu/127.23.115.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.115.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980676253068-24012-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.115.1
--webserver_port=0
--tserver_master_addrs=127.23.115.62:38609
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.115.1
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980676253068-24012-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:38:27.034308 27662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:27.035604 27662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:27.048729 27667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:27.048753 27668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:27.048828 27670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:27.049391 27662 server_base.cc:1034] running on GCE node
I20260620 18:38:27.049988 27662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:27.051041 27662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:27.052278 27662 hybrid_clock.cc:648] HybridClock initialized: now 1781980707052195 us; error 59 us; skew 500 ppm
I20260620 18:38:27.052616 27662 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:38:27 dist-test-slave-d645 krb5kdc[27556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980707, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:27.055768 27662 init.cc:377] Logged in from keytab as kudu/127.23.115.1@KRBTEST.COM (short username kudu)
I20260620 18:38:27.057408 27662 webserver.cc:460] Webserver started at http://127.23.115.1:45299/ using document root <none> and password file <none>
I20260620 18:38:27.058063 27662 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:27.058161 27662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:27.058421 27662 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:38:27.061512 27662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980676253068-24012-0/minicluster-data/ts-0/data/instance:
uuid: "c0cccdbfd1c045d38eeaaa42caa9e255"
format_stamp: "Formatted at 2026-06-20 18:38:27 on dist-test-slave-d645"
server_key: "274959ec3d4b9c571504b97973164625"
server_key_iv: "f69d327e3e113ec9e426f8398a2edb02"
server_key_version: "encryptionkey@0"
I20260620 18:38:27.062278 27662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980676253068-24012-0/minicluster-data/ts-0/wal/instance:
uuid: "c0cccdbfd1c045d38eeaaa42caa9e255"
format_stamp: "Formatted at 2026-06-20 18:38:27 on dist-test-slave-d645"
server_key: "274959ec3d4b9c571504b97973164625"
server_key_iv: "f69d327e3e113ec9e426f8398a2edb02"
server_key_version: "encryptionkey@0"
I20260620 18:38:27.067451 27662 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260620 18:38:27.070811 27677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:27.072168 27662 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:38:27.072319 27662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980676253068-24012-0/minicluster-data/ts-0/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980676253068-24012-0/minicluster-data/ts-0/wal
uuid: "c0cccdbfd1c045d38eeaaa42caa9e255"
format_stamp: "Formatted at 2026-06-20 18:38:27 on dist-test-slave-d645"
server_key: "274959ec3d4b9c571504b97973164625"
server_key_iv: "f69d327e3e113ec9e426f8398a2edb02"
server_key_version: "encryptionkey@0"
I20260620 18:38:27.072516 27662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980676253068-24012-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980676253068-24012-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980676253068-24012-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:27.084326 27662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:27.089008 27662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:27.089324 27662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:27.090325 27662 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:38:27.091645 27662 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:38:27.091713 27662 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:27.091805 27662 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:38:27.091846 27662 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:27.123962 27662 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.1:42277
I20260620 18:38:27.123996 27789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.1:42277 every 8 connection(s)
I20260620 18:38:27.125146 27662 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980676253068-24012-0/minicluster-data/ts-0/data/info.pb
I20260620 18:38:27.126624 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 27662
I20260620 18:38:27.126799 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980676253068-24012-0/minicluster-data/ts-0/wal/instance
I20260620 18:38:27.127887 24012 external_mini_cluster.cc:1428] Setting key 0d6373c61761b67d3f2e9353593c6c0f
Jun 20 18:38:27 dist-test-slave-d645 krb5kdc[27556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980707, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.1@KRBTEST.COM for kudu/127.23.115.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.115.2@KRBTEST.COM; defaulting to no policy
I20260620 18:38:27.146582 27792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:27.127764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.1:52081 (local address 127.23.115.62:38609)
0620 18:38:27.128176 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:27.128180 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:27.129487 (+ 1307us) server_negotiation.cc:407] Connection header received
0620 18:38:27.131496 (+ 2009us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:27.131502 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:27.131581 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:27.131689 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:27.133207 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:27.134158 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:27.136305 (+ 2147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:27.136708 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:27.136827 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:27.140500 (+ 3673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:27.140506 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:27.140509 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:27.140543 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:27.143610 (+ 3067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:27.144336 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:27.144340 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:27.144342 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:27.144464 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:27.144873 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:27.144876 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:27.144878 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:27.145171 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:27.145320 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:27.146081 (+ 761us) server_negotiation.cc:299] Negotiation successful
0620 18:38:27.146306 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":221,"threads_started":1}
I20260620 18:38:27.147678 27793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:27.128436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:38609 (local address 127.23.115.1:52081)
0620 18:38:27.129311 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:27.129364 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:27.131228 (+ 1864us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:27.131893 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:27.131901 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:27.132475 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:27.133041 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:27.133059 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:27.134346 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:27.134351 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:27.136155 (+ 1804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:27.136176 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:27.136981 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:27.136990 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:27.137194 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:27.137856 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:27.137877 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:27.140262 (+ 2385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:27.143755 (+ 3493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:27.143765 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:27.143783 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:27.144211 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:27.144566 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:27.144572 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:27.144575 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:27.144762 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:27.145381 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:27.145388 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:27.145715 (+ 327us) client_negotiation.cc:769] Sending connection context
0620 18:38:27.145954 (+ 239us) client_negotiation.cc:240] Negotiation successful
0620 18:38:27.146374 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":368,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.115.2@KRBTEST.COM" created.
I20260620 18:38:27.149263 27790 heartbeater.cc:347] Connected to a master server at 127.23.115.62:38609
I20260620 18:38:27.149561 27790 heartbeater.cc:464] Registering TS with master...
I20260620 18:38:27.150269 27790 heartbeater.cc:511] Master 127.23.115.62:38609 requested a full tablet report, sending...
I20260620 18:38:27.152204 27601 ts_manager.cc:194] Registered new tserver with Master: c0cccdbfd1c045d38eeaaa42caa9e255 (127.23.115.1:42277)
I20260620 18:38:27.153549 27601 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.115.1@KRBTEST.COM'} at 127.23.115.1:52081
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.115.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.2.keytab.
Entry for principal kudu/127.23.115.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.2.keytab.
I20260620 18:38:27.169701 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980676253068-24012-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980676253068-24012-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980676253068-24012-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980676253068-24012-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.115.2:0
--local_ip_for_outbound_sockets=127.23.115.2
--webserver_interface=127.23.115.2
--webserver_port=0
--tserver_master_addrs=127.23.115.62:38609
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.2.keytab
--principal=kudu/127.23.115.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:37099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:38:27.492316 27796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:27.492576 27796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:27.492666 27796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:27.499827 27796 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:27.499931 27796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:27.500041 27796 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.115.2
I20260620 18:38:27.509872 27796 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:37099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980676253068-24012-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980676253068-24012-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.2.keytab
--principal=kudu/127.23.115.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.115.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980676253068-24012-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.115.2
--webserver_port=0
--tserver_master_addrs=127.23.115.62:38609
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.115.2
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980676253068-24012-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:38:27.511085 27796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:27.512336 27796 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:27.525044 27802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:27.525202 27801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:27.525226 27804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:27.525658 27796 server_base.cc:1034] running on GCE node
I20260620 18:38:27.526269 27796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:27.527185 27796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:27.528398 27796 hybrid_clock.cc:648] HybridClock initialized: now 1781980707528342 us; error 40 us; skew 500 ppm
I20260620 18:38:27.528707 27796 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:38:27 dist-test-slave-d645 krb5kdc[27556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980707, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:27.531709 27796 init.cc:377] Logged in from keytab as kudu/127.23.115.2@KRBTEST.COM (short username kudu)
I20260620 18:38:27.533277 27796 webserver.cc:460] Webserver started at http://127.23.115.2:42783/ using document root <none> and password file <none>
I20260620 18:38:27.533931 27796 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:27.534057 27796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:27.534593 27796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:38:27.537672 27796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980676253068-24012-0/minicluster-data/ts-1/data/instance:
uuid: "0aa6eb71842a44de9aa75fb6ae059a43"
format_stamp: "Formatted at 2026-06-20 18:38:27 on dist-test-slave-d645"
server_key: "38fe712d338f1d520760552abc8c33fb"
server_key_iv: "8cb5dabcef2533b69742824116d4f5a5"
server_key_version: "encryptionkey@0"
I20260620 18:38:27.538407 27796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980676253068-24012-0/minicluster-data/ts-1/wal/instance:
uuid: "0aa6eb71842a44de9aa75fb6ae059a43"
format_stamp: "Formatted at 2026-06-20 18:38:27 on dist-test-slave-d645"
server_key: "38fe712d338f1d520760552abc8c33fb"
server_key_iv: "8cb5dabcef2533b69742824116d4f5a5"
server_key_version: "encryptionkey@0"
I20260620 18:38:27.543476 27796 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260620 18:38:27.546737 27811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:27.548013 27796 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:38:27.548158 27796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980676253068-24012-0/minicluster-data/ts-1/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980676253068-24012-0/minicluster-data/ts-1/wal
uuid: "0aa6eb71842a44de9aa75fb6ae059a43"
format_stamp: "Formatted at 2026-06-20 18:38:27 on dist-test-slave-d645"
server_key: "38fe712d338f1d520760552abc8c33fb"
server_key_iv: "8cb5dabcef2533b69742824116d4f5a5"
server_key_version: "encryptionkey@0"
I20260620 18:38:27.548341 27796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980676253068-24012-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980676253068-24012-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980676253068-24012-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:27.565932 27796 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:27.570755 27796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:27.571102 27796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:27.572194 27796 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:38:27.573751 27796 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:38:27.573848 27796 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:27.573980 27796 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:38:27.574043 27796 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:27.606428 27796 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.2:46533
I20260620 18:38:27.606472 27923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.2:46533 every 8 connection(s)
I20260620 18:38:27.607653 27796 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980676253068-24012-0/minicluster-data/ts-1/data/info.pb
I20260620 18:38:27.617312 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 27796
I20260620 18:38:27.617501 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980676253068-24012-0/minicluster-data/ts-1/wal/instance
I20260620 18:38:27.618062 24012 external_mini_cluster.cc:1428] Setting key 12d45b0719a537782d4a7f0096a619d1
Jun 20 18:38:27 dist-test-slave-d645 krb5kdc[27556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980707, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.2@KRBTEST.COM for kudu/127.23.115.62@KRBTEST.COM
I20260620 18:38:27.627203 27792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:27.610062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.2:43631 (local address 127.23.115.62:38609)
0620 18:38:27.610236 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:27.610240 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:27.611369 (+ 1129us) server_negotiation.cc:407] Connection header received
0620 18:38:27.613112 (+ 1743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:27.613117 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:27.613196 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:27.613281 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:27.614690 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:27.615436 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:27.616994 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:27.617380 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:27.617486 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:27.621357 (+ 3871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:27.621362 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:27.621365 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:27.621399 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:27.623968 (+ 2569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:27.624646 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:27.624651 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:27.624653 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:27.624777 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:27.625249 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:27.625255 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:27.625258 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:27.625621 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:27.625737 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:27.626553 (+ 816us) server_negotiation.cc:299] Negotiation successful
0620 18:38:27.626797 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 18:38:27.628062 27926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:27.610471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:38609 (local address 127.23.115.2:43631)
0620 18:38:27.611217 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:27.611258 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:27.612867 (+ 1609us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:27.613510 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:27.613518 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:27.614007 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:27.614534 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:27.614550 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:27.615577 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:27.615581 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:27.616841 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:27.616853 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:27.617663 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:27.617671 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:27.617977 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:27.618628 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:27.618652 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:27.621150 (+ 2498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:27.624101 (+ 2951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:27.624110 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:27.624126 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:27.624534 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:27.624877 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:27.624882 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:27.624886 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:27.625120 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:27.625737 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:27.625745 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:27.626155 (+ 410us) client_negotiation.cc:769] Sending connection context
0620 18:38:27.626394 (+ 239us) client_negotiation.cc:240] Negotiation successful
0620 18:38:27.626841 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":280,"threads_started":1}
I20260620 18:38:27.629453 27924 heartbeater.cc:347] Connected to a master server at 127.23.115.62:38609
I20260620 18:38:27.629838 27924 heartbeater.cc:464] Registering TS with master...
I20260620 18:38:27.630692 27924 heartbeater.cc:511] Master 127.23.115.62:38609 requested a full tablet report, sending...
I20260620 18:38:27.632432 27601 ts_manager.cc:194] Registered new tserver with Master: 0aa6eb71842a44de9aa75fb6ae059a43 (127.23.115.2:46533)
I20260620 18:38:27.633567 27601 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.115.2@KRBTEST.COM'} at 127.23.115.2:43631
WARNING: no policy specified for kudu/127.23.115.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.115.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.115.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.3.keytab.
Entry for principal kudu/127.23.115.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.3.keytab.
I20260620 18:38:27.663232 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980676253068-24012-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980676253068-24012-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980676253068-24012-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980676253068-24012-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.115.3:0
--local_ip_for_outbound_sockets=127.23.115.3
--webserver_interface=127.23.115.3
--webserver_port=0
--tserver_master_addrs=127.23.115.62:38609
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.3.keytab
--principal=kudu/127.23.115.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:37099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:38:27.970985 27929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:27.971271 27929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:27.971403 27929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:27.979280 27929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:27.979403 27929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:27.979547 27929 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.115.3
I20260620 18:38:27.989746 27929 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:37099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980676253068-24012-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980676253068-24012-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.3.keytab
--principal=kudu/127.23.115.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.115.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980676253068-24012-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.115.3
--webserver_port=0
--tserver_master_addrs=127.23.115.62:38609
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.115.3
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980676253068-24012-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:38:27.991132 27929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:27.993224 27929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:28.007596 27934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:28.007884 27935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:28.008040 27937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:28.008921 27929 server_base.cc:1034] running on GCE node
I20260620 18:38:28.009672 27929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:28.011119 27929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:28.012527 27929 hybrid_clock.cc:648] HybridClock initialized: now 1781980708012449 us; error 29 us; skew 500 ppm
I20260620 18:38:28.013024 27929 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:38:28 dist-test-slave-d645 krb5kdc[27556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980708, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:28.017153 27929 init.cc:377] Logged in from keytab as kudu/127.23.115.3@KRBTEST.COM (short username kudu)
I20260620 18:38:28.018991 27929 webserver.cc:460] Webserver started at http://127.23.115.3:46781/ using document root <none> and password file <none>
I20260620 18:38:28.019752 27929 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:28.019891 27929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:28.020226 27929 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:38:28.023407 27929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980676253068-24012-0/minicluster-data/ts-2/data/instance:
uuid: "b7ba651a6e7647eb91904b5e381a09eb"
format_stamp: "Formatted at 2026-06-20 18:38:28 on dist-test-slave-d645"
server_key: "7d57265f39c4d91d629e92e54999be1c"
server_key_iv: "b2c308addb4705966d92103ca174de5a"
server_key_version: "encryptionkey@0"
I20260620 18:38:28.024245 27929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980676253068-24012-0/minicluster-data/ts-2/wal/instance:
uuid: "b7ba651a6e7647eb91904b5e381a09eb"
format_stamp: "Formatted at 2026-06-20 18:38:28 on dist-test-slave-d645"
server_key: "7d57265f39c4d91d629e92e54999be1c"
server_key_iv: "b2c308addb4705966d92103ca174de5a"
server_key_version: "encryptionkey@0"
I20260620 18:38:28.030618 27929 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260620 18:38:28.034228 27944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:28.035861 27929 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:38:28.036038 27929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980676253068-24012-0/minicluster-data/ts-2/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980676253068-24012-0/minicluster-data/ts-2/wal
uuid: "b7ba651a6e7647eb91904b5e381a09eb"
format_stamp: "Formatted at 2026-06-20 18:38:28 on dist-test-slave-d645"
server_key: "7d57265f39c4d91d629e92e54999be1c"
server_key_iv: "b2c308addb4705966d92103ca174de5a"
server_key_version: "encryptionkey@0"
I20260620 18:38:28.036244 27929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980676253068-24012-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980676253068-24012-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980676253068-24012-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:28.049449 27929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:28.053783 27929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:28.054054 27929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:28.055083 27929 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:38:28.056424 27929 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:38:28.056502 27929 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:28.056592 27929 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:38:28.056633 27929 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:28.085326 27929 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.3:46669
I20260620 18:38:28.085403 28056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.3:46669 every 8 connection(s)
I20260620 18:38:28.086539 27929 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980676253068-24012-0/minicluster-data/ts-2/data/info.pb
I20260620 18:38:28.090389 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 27929
I20260620 18:38:28.090610 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980676253068-24012-0/minicluster-data/ts-2/wal/instance
I20260620 18:38:28.091154 24012 external_mini_cluster.cc:1428] Setting key 577d0c7513eef33748b4b8cf63b39436
Jun 20 18:38:28 dist-test-slave-d645 krb5kdc[27556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980708, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.3@KRBTEST.COM for kudu/127.23.115.62@KRBTEST.COM
I20260620 18:38:28.108553 27792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:28.089292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.3:54263 (local address 127.23.115.62:38609)
0620 18:38:28.089468 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:28.089473 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:28.090699 (+ 1226us) server_negotiation.cc:407] Connection header received
0620 18:38:28.093069 (+ 2370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:28.093073 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:28.093149 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:28.093239 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:28.094814 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:28.095608 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:28.097232 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:28.097732 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:28.097853 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:28.102377 (+ 4524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:28.102382 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:28.102386 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:28.102419 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:28.105131 (+ 2712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:28.106230 (+ 1099us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:28.106236 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:28.106240 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:28.106416 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:28.106967 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:28.106980 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:28.106984 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:28.107277 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:28.107389 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:28.108120 (+ 731us) server_negotiation.cc:299] Negotiation successful
0620 18:38:28.108297 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 18:38:28.109553 28059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:28.089726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:38609 (local address 127.23.115.3:54263)
0620 18:38:28.090525 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:28.090591 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:28.092779 (+ 2188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:28.093447 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:28.093455 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:28.094037 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:28.094661 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:28.094678 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:28.095740 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:28.095744 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:28.097089 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:28.097101 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:28.098040 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:28.098049 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:28.098362 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:28.099455 (+ 1093us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:28.099485 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:28.102139 (+ 2654us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:28.105280 (+ 3141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:28.105287 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:28.105300 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:28.106027 (+ 727us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:28.106561 (+ 534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:28.106566 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:28.106575 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:28.106818 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:28.107388 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:28.107394 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:28.107770 (+ 376us) client_negotiation.cc:769] Sending connection context
0620 18:38:28.108055 (+ 285us) client_negotiation.cc:240] Negotiation successful
0620 18:38:28.108488 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":297,"threads_started":1}
I20260620 18:38:28.110905 28057 heartbeater.cc:347] Connected to a master server at 127.23.115.62:38609
I20260620 18:38:28.111275 28057 heartbeater.cc:464] Registering TS with master...
I20260620 18:38:28.111972 28057 heartbeater.cc:511] Master 127.23.115.62:38609 requested a full tablet report, sending...
I20260620 18:38:28.113309 27601 ts_manager.cc:194] Registered new tserver with Master: b7ba651a6e7647eb91904b5e381a09eb (127.23.115.3:46669)
I20260620 18:38:28.114068 27601 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.115.3@KRBTEST.COM'} at 127.23.115.3:54263
I20260620 18:38:28.119046 24012 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 18:38:28 dist-test-slave-d645 krb5kdc[27556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980708, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
I20260620 18:38:28.156351 27790 heartbeater.cc:503] Master 127.23.115.62:38609 was elected leader, sending a full tablet report...
Jun 20 18:38:28 dist-test-slave-d645 krb5kdc[27556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980708, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.115.1@KRBTEST.COM
I20260620 18:38:28.165575 28064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:28.150616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44468 (local address 127.23.115.1:42277)
0620 18:38:28.151231 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:28.151238 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:28.151266 (+ 28us) server_negotiation.cc:407] Connection header received
0620 18:38:28.151330 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:28.151336 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:28.151577 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:28.151738 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:28.153159 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:28.154214 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:28.155454 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:28.156028 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:28.156134 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:28.160076 (+ 3942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:28.160087 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:28.160102 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:28.160139 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:28.162851 (+ 2712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:28.163580 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:28.163588 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:28.163590 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:28.163715 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:28.164073 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:28.164076 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:28.164078 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:28.164418 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:28.164693 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:28.164992 (+ 299us) server_negotiation.cc:299] Negotiation successful
0620 18:38:28.165251 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":337,"threads_started":1}
Jun 20 18:38:28 dist-test-slave-d645 krb5kdc[27556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980708, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.115.62@KRBTEST.COM
I20260620 18:38:28.182849 27792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:28.169576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58740 (local address 127.23.115.62:38609)
0620 18:38:28.169802 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:28.169807 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:28.169906 (+ 99us) server_negotiation.cc:407] Connection header received
0620 18:38:28.170026 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:28.170030 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:28.170090 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:28.170165 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:28.171342 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:28.172053 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:28.173213 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:28.173614 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:28.173701 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:28.177755 (+ 4054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:28.177760 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:28.177764 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:28.177798 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:28.180310 (+ 2512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:28.180949 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:28.180953 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:28.180955 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:28.181077 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:28.181407 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:28.181411 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:28.181413 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:28.181746 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:28.181924 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:28.182279 (+ 355us) server_negotiation.cc:299] Negotiation successful
0620 18:38:28.182449 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
W20260620 18:38:28.183506 27601 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58740
I20260620 18:38:28.184451 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 27662
I20260620 18:38:28.198693 28064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:28.185271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44472 (local address 127.23.115.1:42277)
0620 18:38:28.185437 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:28.185441 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:28.185785 (+ 344us) server_negotiation.cc:407] Connection header received
0620 18:38:28.185920 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:28.185924 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:28.185985 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:28.186054 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:28.187148 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:28.187824 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:28.189165 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:28.189592 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:28.189700 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:28.192075 (+ 2375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:28.192081 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:28.192084 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:28.192115 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:28.195355 (+ 3240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:28.196243 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:28.196248 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:28.196251 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:28.196492 (+ 241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:28.197029 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:28.197035 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:28.197039 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:28.197454 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:28.197782 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:28.198157 (+ 375us) server_negotiation.cc:299] Negotiation successful
0620 18:38:28.198387 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 18:38:28.199340 27785 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:38:28.281538 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 27662
I20260620 18:38:28.294761 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 27796
Jun 20 18:38:28 dist-test-slave-d645 krb5kdc[27556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980708, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.115.2@KRBTEST.COM
I20260620 18:38:28.311352 28069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:28.295673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47344 (local address 127.23.115.2:46533)
0620 18:38:28.296126 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:28.296133 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:28.296157 (+ 24us) server_negotiation.cc:407] Connection header received
0620 18:38:28.296224 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:28.296231 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:28.296484 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:28.296601 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:28.297970 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:28.299041 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:28.300305 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:28.300850 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:28.300962 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:28.305479 (+ 4517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:28.305501 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:28.305522 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:28.305579 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:28.308745 (+ 3166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:28.309444 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:28.309453 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:28.309456 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:28.309594 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:28.309930 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:28.309934 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:28.309936 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:28.310281 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:28.310523 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:28.310817 (+ 294us) server_negotiation.cc:299] Negotiation successful
0620 18:38:28.311069 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":210,"threads_started":1}
I20260620 18:38:28.312621 27919 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:38:28.393489 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 27796
I20260620 18:38:28.406205 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 27929
Jun 20 18:38:28 dist-test-slave-d645 krb5kdc[27556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980708, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.115.3@KRBTEST.COM
I20260620 18:38:28.422521 28071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:28.407204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55148 (local address 127.23.115.3:46669)
0620 18:38:28.407829 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:28.407836 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:28.407863 (+ 27us) server_negotiation.cc:407] Connection header received
0620 18:38:28.407920 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:28.407926 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:28.408194 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:28.408318 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:28.409574 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:28.410610 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:28.412122 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:28.412720 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:28.412843 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:28.417030 (+ 4187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:28.417043 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:28.417060 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:28.417098 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:28.419831 (+ 2733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:28.420482 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:28.420490 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:28.420492 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:28.420606 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:28.420889 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:28.420892 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:28.420894 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:28.421333 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:28.421579 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:28.421920 (+ 341us) server_negotiation.cc:299] Negotiation successful
0620 18:38:28.422129 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":315,"threads_started":1}
I20260620 18:38:28.423875 28052 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:38:28.504685 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 27929
I20260620 18:38:28.517565 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 27569
I20260620 18:38:28.518771 27631 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:38:28.620256 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 27569
2026-06-20T18:38:28Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3694 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980676253068-24012-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:38:28 dist-test-slave-d645 krb5kdc[28076](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:38:28 dist-test-slave-d645 krb5kdc[28076](info): set up 2 sockets
Jun 20 18:38:28 dist-test-slave-d645 krb5kdc[28076](info): commencing operation
krb5kdc: starting...
W20260620 18:38:29.635536 24012 mini_kdc.cc:121] Time spent starting KDC: real 0.977s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980676253068-24012-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980676253068-24012-0/krb5kdc/test-user.keytab.
Jun 20 18:38:29 dist-test-slave-d645 krb5kdc[28076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980709, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:38:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:38:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.115.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.115.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.115.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab.
Entry for principal kudu/127.23.115.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab.
I20260620 18:38:29.820501 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980676253068-24012-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980676253068-24012-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980676253068-24012-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.115.62:36185
--webserver_interface=127.23.115.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab
--principal=kudu/127.23.115.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:44175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.115.62:36185
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:38:30.128285 28089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:30.128574 28089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:30.128695 28089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:30.136158 28089 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:38:30.136268 28089 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:30.136339 28089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:30.136420 28089 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:38:30.136478 28089 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:38:30.146327 28089 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:44175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980676253068-24012-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980676253068-24012-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.115.62:36185
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab
--principal=kudu/127.23.115.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.115.62:36185
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.115.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980676253068-24012-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:38:30.147620 28089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:30.148875 28089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:30.159879 28095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:30.160180 28094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:30.160156 28097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:30.161787 28089 server_base.cc:1034] running on GCE node
I20260620 18:38:30.162653 28089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:30.164330 28089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:30.165550 28089 hybrid_clock.cc:648] HybridClock initialized: now 1781980710165494 us; error 58 us; skew 500 ppm
I20260620 18:38:30.165900 28089 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:38:30 dist-test-slave-d645 krb5kdc[28076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980710, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:30.169132 28089 init.cc:377] Logged in from keytab as kudu/127.23.115.62@KRBTEST.COM (short username kudu)
I20260620 18:38:30.170264 28089 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:30.170352 28089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:30.170635 28089 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:38:30.173527 28089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980676253068-24012-0/minicluster-data/master-0/data/instance:
uuid: "b9c52c861bc34190bc131f82443df04a"
format_stamp: "Formatted at 2026-06-20 18:38:30 on dist-test-slave-d645"
server_key: "1c5e88171714201ae116dc0caedc357b"
server_key_iv: "31f07700798fc0dc38e3e5f8d5891bf1"
server_key_version: "encryptionkey@0"
I20260620 18:38:30.174291 28089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980676253068-24012-0/minicluster-data/master-0/wal/instance:
uuid: "b9c52c861bc34190bc131f82443df04a"
format_stamp: "Formatted at 2026-06-20 18:38:30 on dist-test-slave-d645"
server_key: "1c5e88171714201ae116dc0caedc357b"
server_key_iv: "31f07700798fc0dc38e3e5f8d5891bf1"
server_key_version: "encryptionkey@0"
I20260620 18:38:30.179276 28089 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 18:38:30.182624 28103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:30.183967 28089 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:38:30.184139 28089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980676253068-24012-0/minicluster-data/master-0/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980676253068-24012-0/minicluster-data/master-0/wal
uuid: "b9c52c861bc34190bc131f82443df04a"
format_stamp: "Formatted at 2026-06-20 18:38:30 on dist-test-slave-d645"
server_key: "1c5e88171714201ae116dc0caedc357b"
server_key_iv: "31f07700798fc0dc38e3e5f8d5891bf1"
server_key_version: "encryptionkey@0"
I20260620 18:38:30.184338 28089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980676253068-24012-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980676253068-24012-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980676253068-24012-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:30.197415 28089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:30.201193 28089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:30.201434 28089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:30.219028 28089 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.62:36185
I20260620 18:38:30.219069 28154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.62:36185 every 8 connection(s)
I20260620 18:38:30.219866 28089 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
I20260620 18:38:30.223430 28155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:30.227819 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 28089
I20260620 18:38:30.228031 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980676253068-24012-0/minicluster-data/master-0/wal/instance
I20260620 18:38:30.228622 24012 external_mini_cluster.cc:1428] Setting key 3674a23d3d3e0a30cb3cf62684f61f51
I20260620 18:38:30.234515 28155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9c52c861bc34190bc131f82443df04a: Bootstrap starting.
I20260620 18:38:30.238271 28155 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b9c52c861bc34190bc131f82443df04a: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:30.239494 28155 log.cc:824] T 00000000000000000000000000000000 P b9c52c861bc34190bc131f82443df04a: Log is configured to *not* fsync() on all Append() calls
I20260620 18:38:30.242673 28155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9c52c861bc34190bc131f82443df04a: No bootstrap required, opened a new log
Jun 20 18:38:30 dist-test-slave-d645 krb5kdc[28076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980709, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.115.62@KRBTEST.COM
I20260620 18:38:30.250360 28155 raft_consensus.cc:348] T 00000000000000000000000000000000 P b9c52c861bc34190bc131f82443df04a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9c52c861bc34190bc131f82443df04a" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 36185 } }
I20260620 18:38:30.250742 28155 raft_consensus.cc:374] T 00000000000000000000000000000000 P b9c52c861bc34190bc131f82443df04a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:30.250830 28155 raft_consensus.cc:729] T 00000000000000000000000000000000 P b9c52c861bc34190bc131f82443df04a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9c52c861bc34190bc131f82443df04a, State: Initialized, Role: FOLLOWER
I20260620 18:38:30.251169 28158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:30.230488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56696 (local address 127.23.115.62:36185)
0620 18:38:30.231304 (+ 816us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:30.231315 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:30.231370 (+ 55us) server_negotiation.cc:407] Connection header received
0620 18:38:30.233057 (+ 1687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:30.233077 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:30.233430 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:30.233720 (+ 290us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:30.235515 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:30.236975 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:30.238724 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:30.239430 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:30.239569 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:30.244007 (+ 4438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:30.244025 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:30.244043 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:30.244085 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:30.247158 (+ 3073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:30.247806 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:30.247812 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:30.247818 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:30.247999 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:30.248344 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:30.248347 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:30.248349 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:30.248907 (+ 558us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:30.249194 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:30.249626 (+ 432us) server_negotiation.cc:299] Negotiation successful
0620 18:38:30.250036 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":398,"threads_started":1}
I20260620 18:38:30.251523 28155 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9c52c861bc34190bc131f82443df04a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9c52c861bc34190bc131f82443df04a" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 36185 } }
I20260620 18:38:30.251806 28155 raft_consensus.cc:388] T 00000000000000000000000000000000 P b9c52c861bc34190bc131f82443df04a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:38:30.251945 28155 raft_consensus.cc:482] T 00000000000000000000000000000000 P b9c52c861bc34190bc131f82443df04a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:38:30.252120 28155 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b9c52c861bc34190bc131f82443df04a [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:30.255076 28155 raft_consensus.cc:504] T 00000000000000000000000000000000 P b9c52c861bc34190bc131f82443df04a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9c52c861bc34190bc131f82443df04a" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 36185 } }
I20260620 18:38:30.255519 28155 leader_election.cc:302] T 00000000000000000000000000000000 P b9c52c861bc34190bc131f82443df04a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b9c52c861bc34190bc131f82443df04a; no voters:
I20260620 18:38:30.256072 28155 leader_election.cc:288] T 00000000000000000000000000000000 P b9c52c861bc34190bc131f82443df04a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:38:30.256207 28160 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b9c52c861bc34190bc131f82443df04a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:38:30.256899 28160 raft_consensus.cc:686] T 00000000000000000000000000000000 P b9c52c861bc34190bc131f82443df04a [term 1 LEADER]: Becoming Leader. State: Replica: b9c52c861bc34190bc131f82443df04a, State: Running, Role: LEADER
I20260620 18:38:30.257431 28160 consensus_queue.cc:237] T 00000000000000000000000000000000 P b9c52c861bc34190bc131f82443df04a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9c52c861bc34190bc131f82443df04a" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 36185 } }
I20260620 18:38:30.258263 28155 sys_catalog.cc:564] T 00000000000000000000000000000000 P b9c52c861bc34190bc131f82443df04a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:38:30.260136 28162 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9c52c861bc34190bc131f82443df04a [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9c52c861bc34190bc131f82443df04a. Latest consensus state: current_term: 1 leader_uuid: "b9c52c861bc34190bc131f82443df04a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9c52c861bc34190bc131f82443df04a" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 36185 } } }
I20260620 18:38:30.260496 28162 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9c52c861bc34190bc131f82443df04a [sys.catalog]: This master's current role is: LEADER
I20260620 18:38:30.260969 28161 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9c52c861bc34190bc131f82443df04a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b9c52c861bc34190bc131f82443df04a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9c52c861bc34190bc131f82443df04a" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 36185 } } }
I20260620 18:38:30.261276 28161 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9c52c861bc34190bc131f82443df04a [sys.catalog]: This master's current role is: LEADER
I20260620 18:38:30.263567 28167 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:38:30.268996 28167 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:38:30.278231 28167 catalog_manager.cc:1347] Generated new cluster ID: 90ce94f6df5743b8a398b18f353f70ed
I20260620 18:38:30.278365 28167 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:38:30.288880 28167 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:38:30.290278 28167 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:38:30.299180 28167 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b9c52c861bc34190bc131f82443df04a: Generated new TSK 0
I20260620 18:38:30.300047 28167 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.115.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.115.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.115.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.1.keytab.
Entry for principal kudu/127.23.115.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.1.keytab.
I20260620 18:38:30.352829 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980676253068-24012-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980676253068-24012-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980676253068-24012-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980676253068-24012-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.115.1:0
--local_ip_for_outbound_sockets=127.23.115.1
--webserver_interface=127.23.115.1
--webserver_port=0
--tserver_master_addrs=127.23.115.62:36185
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.1.keytab
--principal=kudu/127.23.115.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:44175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:38:30.661201 28181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:30.661464 28181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:30.661579 28181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:30.668876 28181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:30.668977 28181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:30.669127 28181 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.115.1
I20260620 18:38:30.678552 28181 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:44175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980676253068-24012-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980676253068-24012-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.1.keytab
--principal=kudu/127.23.115.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.115.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980676253068-24012-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.115.1
--webserver_port=0
--tserver_master_addrs=127.23.115.62:36185
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.115.1
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980676253068-24012-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:38:30.679936 28181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:30.681185 28181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:30.693287 28189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:30.693362 28187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:30.693382 28186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:30.694283 28181 server_base.cc:1034] running on GCE node
I20260620 18:38:30.694866 28181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:30.695798 28181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:30.697019 28181 hybrid_clock.cc:648] HybridClock initialized: now 1781980710696962 us; error 49 us; skew 500 ppm
I20260620 18:38:30.697330 28181 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:38:30 dist-test-slave-d645 krb5kdc[28076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980710, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:30.700350 28181 init.cc:377] Logged in from keytab as kudu/127.23.115.1@KRBTEST.COM (short username kudu)
I20260620 18:38:30.701370 28181 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:30.701463 28181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:30.701709 28181 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:38:30.704483 28181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980676253068-24012-0/minicluster-data/ts-0/data/instance:
uuid: "9aa158f5757f44b6a00beb2c61417931"
format_stamp: "Formatted at 2026-06-20 18:38:30 on dist-test-slave-d645"
server_key: "a817a1d70d5ed0c5ca5be21e6678c583"
server_key_iv: "c1a9a9ee01dd9f1211feca2db4583ceb"
server_key_version: "encryptionkey@0"
I20260620 18:38:30.705420 28181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980676253068-24012-0/minicluster-data/ts-0/wal/instance:
uuid: "9aa158f5757f44b6a00beb2c61417931"
format_stamp: "Formatted at 2026-06-20 18:38:30 on dist-test-slave-d645"
server_key: "a817a1d70d5ed0c5ca5be21e6678c583"
server_key_iv: "c1a9a9ee01dd9f1211feca2db4583ceb"
server_key_version: "encryptionkey@0"
I20260620 18:38:30.711829 28181 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260620 18:38:30.715343 28195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:30.716682 28181 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 18:38:30.716838 28181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980676253068-24012-0/minicluster-data/ts-0/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980676253068-24012-0/minicluster-data/ts-0/wal
uuid: "9aa158f5757f44b6a00beb2c61417931"
format_stamp: "Formatted at 2026-06-20 18:38:30 on dist-test-slave-d645"
server_key: "a817a1d70d5ed0c5ca5be21e6678c583"
server_key_iv: "c1a9a9ee01dd9f1211feca2db4583ceb"
server_key_version: "encryptionkey@0"
I20260620 18:38:30.717041 28181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980676253068-24012-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980676253068-24012-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980676253068-24012-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:30.740746 28181 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:30.744787 28181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:30.745034 28181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:30.745965 28181 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:38:30.747308 28181 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:38:30.747397 28181 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:30.747498 28181 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:38:30.747567 28181 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:30.778184 28181 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.1:37551
I20260620 18:38:30.778230 28307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.1:37551 every 8 connection(s)
I20260620 18:38:30.779070 28181 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980676253068-24012-0/minicluster-data/ts-0/data/info.pb
I20260620 18:38:30.781844 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 28181
I20260620 18:38:30.782060 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980676253068-24012-0/minicluster-data/ts-0/wal/instance
I20260620 18:38:30.782675 24012 external_mini_cluster.cc:1428] Setting key 823d8bfd2774faefe071c8344c52efa9
Jun 20 18:38:30 dist-test-slave-d645 krb5kdc[28076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980710, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.1@KRBTEST.COM for kudu/127.23.115.62@KRBTEST.COM
I20260620 18:38:30.799000 28310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:30.781379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.1:53101 (local address 127.23.115.62:36185)
0620 18:38:30.781878 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:30.781885 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:30.782773 (+ 888us) server_negotiation.cc:407] Connection header received
0620 18:38:30.784575 (+ 1802us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:30.784580 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:30.784655 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:30.784772 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:30.786203 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:30.787192 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:30.788808 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:30.789199 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:30.789308 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:30.792754 (+ 3446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:30.792760 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:30.792764 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:30.792800 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:30.795776 (+ 2976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:30.796512 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:30.796518 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:30.796521 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:30.796659 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:30.797059 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:30.797062 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:30.797065 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:30.797351 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:30.797480 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:30.798360 (+ 880us) server_negotiation.cc:299] Negotiation successful
0620 18:38:30.798595 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":248,"threads_started":1}
WARNING: no policy specified for kudu/127.23.115.2@KRBTEST.COM; defaulting to no policy
I20260620 18:38:30.800103 28311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:30.781805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:36185 (local address 127.23.115.1:53101)
0620 18:38:30.782620 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:30.782669 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:30.784337 (+ 1668us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:30.784993 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:30.785001 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:30.785518 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:30.786035 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:30.786051 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:30.787350 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:30.787354 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:30.788640 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:30.788653 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:30.789414 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:30.789421 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:30.789647 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:30.790256 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:30.790278 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:30.792529 (+ 2251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:30.795938 (+ 3409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:30.795945 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:30.795958 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:30.796391 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:30.796765 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:30.796768 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:30.796770 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:30.796950 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:30.797473 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:30.797481 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:30.797814 (+ 333us) client_negotiation.cc:769] Sending connection context
0620 18:38:30.798219 (+ 405us) client_negotiation.cc:240] Negotiation successful
0620 18:38:30.798747 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":333,"threads_started":1}
I20260620 18:38:30.801719 28308 heartbeater.cc:347] Connected to a master server at 127.23.115.62:36185
I20260620 18:38:30.801993 28308 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.115.2@KRBTEST.COM" created.
I20260620 18:38:30.802680 28308 heartbeater.cc:511] Master 127.23.115.62:36185 requested a full tablet report, sending...
I20260620 18:38:30.804610 28120 ts_manager.cc:194] Registered new tserver with Master: 9aa158f5757f44b6a00beb2c61417931 (127.23.115.1:37551)
I20260620 18:38:30.805974 28120 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.115.1@KRBTEST.COM'} at 127.23.115.1:53101
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.115.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.2.keytab.
Entry for principal kudu/127.23.115.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.2.keytab.
I20260620 18:38:30.826742 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980676253068-24012-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980676253068-24012-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980676253068-24012-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980676253068-24012-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.115.2:0
--local_ip_for_outbound_sockets=127.23.115.2
--webserver_interface=127.23.115.2
--webserver_port=0
--tserver_master_addrs=127.23.115.62:36185
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.2.keytab
--principal=kudu/127.23.115.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:44175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:38:31.143095 28314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:31.143378 28314 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:31.143481 28314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:31.150610 28314 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:31.150719 28314 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:31.150847 28314 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.115.2
I20260620 18:38:31.161135 28314 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:44175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980676253068-24012-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980676253068-24012-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.2.keytab
--principal=kudu/127.23.115.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.115.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980676253068-24012-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.115.2
--webserver_port=0
--tserver_master_addrs=127.23.115.62:36185
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.115.2
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980676253068-24012-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:38:31.162348 28314 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:31.163636 28314 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:31.175871 28320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:31.175933 28319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:31.175958 28322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:31.176555 28314 server_base.cc:1034] running on GCE node
I20260620 18:38:31.177090 28314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:31.177960 28314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:31.179149 28314 hybrid_clock.cc:648] HybridClock initialized: now 1781980711179105 us; error 40 us; skew 500 ppm
I20260620 18:38:31.179440 28314 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:38:31 dist-test-slave-d645 krb5kdc[28076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980711, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:31.182482 28314 init.cc:377] Logged in from keytab as kudu/127.23.115.2@KRBTEST.COM (short username kudu)
I20260620 18:38:31.183609 28314 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:31.183692 28314 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:31.183954 28314 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:38:31.186700 28314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980676253068-24012-0/minicluster-data/ts-1/data/instance:
uuid: "2b6057f1a3ed4e7d939965756ea3c654"
format_stamp: "Formatted at 2026-06-20 18:38:31 on dist-test-slave-d645"
server_key: "2b0aa628eff1b84c35603a1afda89fcc"
server_key_iv: "e0134f59c16599cd33b4844a1490ef54"
server_key_version: "encryptionkey@0"
I20260620 18:38:31.187472 28314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980676253068-24012-0/minicluster-data/ts-1/wal/instance:
uuid: "2b6057f1a3ed4e7d939965756ea3c654"
format_stamp: "Formatted at 2026-06-20 18:38:31 on dist-test-slave-d645"
server_key: "2b0aa628eff1b84c35603a1afda89fcc"
server_key_iv: "e0134f59c16599cd33b4844a1490ef54"
server_key_version: "encryptionkey@0"
I20260620 18:38:31.192252 28314 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 18:38:31.195595 28328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:31.196784 28314 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:38:31.196947 28314 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980676253068-24012-0/minicluster-data/ts-1/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980676253068-24012-0/minicluster-data/ts-1/wal
uuid: "2b6057f1a3ed4e7d939965756ea3c654"
format_stamp: "Formatted at 2026-06-20 18:38:31 on dist-test-slave-d645"
server_key: "2b0aa628eff1b84c35603a1afda89fcc"
server_key_iv: "e0134f59c16599cd33b4844a1490ef54"
server_key_version: "encryptionkey@0"
I20260620 18:38:31.197158 28314 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980676253068-24012-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980676253068-24012-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980676253068-24012-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:31.215124 28314 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:31.219429 28314 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:31.219707 28314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:31.220846 28314 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:38:31.222605 28314 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:38:31.222709 28314 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:31.222816 28314 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:38:31.222883 28314 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:31.253328 28314 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.2:46717
I20260620 18:38:31.253388 28440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.2:46717 every 8 connection(s)
I20260620 18:38:31.254078 28314 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980676253068-24012-0/minicluster-data/ts-1/data/info.pb
I20260620 18:38:31.254350 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 28314
I20260620 18:38:31.254544 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980676253068-24012-0/minicluster-data/ts-1/wal/instance
I20260620 18:38:31.255106 24012 external_mini_cluster.cc:1428] Setting key 01208c02c5db92661f4a1030d782b5e6
Jun 20 18:38:31 dist-test-slave-d645 krb5kdc[28076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980711, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.2@KRBTEST.COM for kudu/127.23.115.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.115.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.115.3@KRBTEST.COM" created.
I20260620 18:38:31.275462 28310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:31.256801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.2:50069 (local address 127.23.115.62:36185)
0620 18:38:31.256987 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:31.256992 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:31.258423 (+ 1431us) server_negotiation.cc:407] Connection header received
0620 18:38:31.260831 (+ 2408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:31.260850 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:31.260965 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:31.261079 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:31.262664 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:31.263555 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:31.265233 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:31.265644 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:31.265749 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:31.269672 (+ 3923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:31.269678 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:31.269682 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:31.269718 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:31.272525 (+ 2807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:31.273237 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:31.273243 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:31.273246 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:31.273367 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:31.273776 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:31.273782 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:31.273785 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:31.274048 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:31.274187 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:31.274860 (+ 673us) server_negotiation.cc:299] Negotiation successful
0620 18:38:31.275132 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 18:38:31.276404 28443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:31.257394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:36185 (local address 127.23.115.2:50069)
0620 18:38:31.258192 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:31.258296 (+ 104us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:31.260289 (+ 1993us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:31.261348 (+ 1059us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:31.261358 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:31.261889 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:31.262467 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:31.262489 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:31.263687 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:31.263691 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:31.265094 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:31.265107 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:31.265853 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:31.265860 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:31.266079 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:31.267103 (+ 1024us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:31.267133 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:31.269397 (+ 2264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:31.272670 (+ 3273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:31.272679 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:31.272696 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:31.273122 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:31.273467 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:31.273472 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:31.273476 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:31.273666 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:31.274170 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:31.274178 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:31.274510 (+ 332us) client_negotiation.cc:769] Sending connection context
0620 18:38:31.274757 (+ 247us) client_negotiation.cc:240] Negotiation successful
0620 18:38:31.275337 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":362,"threads_started":1}
I20260620 18:38:31.277828 28441 heartbeater.cc:347] Connected to a master server at 127.23.115.62:36185
I20260620 18:38:31.278187 28441 heartbeater.cc:464] Registering TS with master...
I20260620 18:38:31.279027 28441 heartbeater.cc:511] Master 127.23.115.62:36185 requested a full tablet report, sending...
I20260620 18:38:31.280735 28120 ts_manager.cc:194] Registered new tserver with Master: 2b6057f1a3ed4e7d939965756ea3c654 (127.23.115.2:46717)
I20260620 18:38:31.281883 28120 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.115.2@KRBTEST.COM'} at 127.23.115.2:50069
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.115.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.3.keytab.
Entry for principal kudu/127.23.115.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.3.keytab.
I20260620 18:38:31.299686 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980676253068-24012-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980676253068-24012-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980676253068-24012-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980676253068-24012-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.115.3:0
--local_ip_for_outbound_sockets=127.23.115.3
--webserver_interface=127.23.115.3
--webserver_port=0
--tserver_master_addrs=127.23.115.62:36185
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.3.keytab
--principal=kudu/127.23.115.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:44175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:38:31.611356 28446 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:31.611639 28446 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:31.611737 28446 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:31.619036 28446 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:31.619148 28446 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:31.619273 28446 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.115.3
I20260620 18:38:31.629035 28446 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:44175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980676253068-24012-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980676253068-24012-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.3.keytab
--principal=kudu/127.23.115.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.115.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980676253068-24012-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.115.3
--webserver_port=0
--tserver_master_addrs=127.23.115.62:36185
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.115.3
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980676253068-24012-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:38:31.630215 28446 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:31.631433 28446 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:31.643867 28452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:31.643867 28451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:31.643889 28454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:31.644671 28446 server_base.cc:1034] running on GCE node
I20260620 18:38:31.645262 28446 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:31.646230 28446 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:31.647455 28446 hybrid_clock.cc:648] HybridClock initialized: now 1781980711647406 us; error 36 us; skew 500 ppm
I20260620 18:38:31.647785 28446 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:38:31 dist-test-slave-d645 krb5kdc[28076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980711, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:31.655084 28446 init.cc:377] Logged in from keytab as kudu/127.23.115.3@KRBTEST.COM (short username kudu)
I20260620 18:38:31.656363 28446 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:31.656495 28446 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:31.656831 28446 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:38:31.660161 28446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980676253068-24012-0/minicluster-data/ts-2/data/instance:
uuid: "415c28c7203044549a4da89e04a99fdc"
format_stamp: "Formatted at 2026-06-20 18:38:31 on dist-test-slave-d645"
server_key: "5037d1d989486d58ed7558200c85e1d3"
server_key_iv: "859a3c07892c6bce968543d60cb6c564"
server_key_version: "encryptionkey@0"
I20260620 18:38:31.661211 28446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980676253068-24012-0/minicluster-data/ts-2/wal/instance:
uuid: "415c28c7203044549a4da89e04a99fdc"
format_stamp: "Formatted at 2026-06-20 18:38:31 on dist-test-slave-d645"
server_key: "5037d1d989486d58ed7558200c85e1d3"
server_key_iv: "859a3c07892c6bce968543d60cb6c564"
server_key_version: "encryptionkey@0"
I20260620 18:38:31.667824 28446 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260620 18:38:31.671221 28460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:31.672528 28446 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 18:38:31.672688 28446 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980676253068-24012-0/minicluster-data/ts-2/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980676253068-24012-0/minicluster-data/ts-2/wal
uuid: "415c28c7203044549a4da89e04a99fdc"
format_stamp: "Formatted at 2026-06-20 18:38:31 on dist-test-slave-d645"
server_key: "5037d1d989486d58ed7558200c85e1d3"
server_key_iv: "859a3c07892c6bce968543d60cb6c564"
server_key_version: "encryptionkey@0"
I20260620 18:38:31.672888 28446 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980676253068-24012-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980676253068-24012-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980676253068-24012-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:31.686697 28446 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:31.690788 28446 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:31.691056 28446 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:31.692020 28446 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:38:31.693307 28446 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:38:31.693396 28446 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:31.693506 28446 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:38:31.693569 28446 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:31.725772 28446 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.3:36497
I20260620 18:38:31.725872 28572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.3:36497 every 8 connection(s)
I20260620 18:38:31.726588 28446 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980676253068-24012-0/minicluster-data/ts-2/data/info.pb
I20260620 18:38:31.727756 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 28446
I20260620 18:38:31.727941 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980676253068-24012-0/minicluster-data/ts-2/wal/instance
I20260620 18:38:31.728327 24012 external_mini_cluster.cc:1428] Setting key 7a1dfbf3a3624772c75f720a26afcbf9
Jun 20 18:38:31 dist-test-slave-d645 krb5kdc[28076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980711, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.3@KRBTEST.COM for kudu/127.23.115.62@KRBTEST.COM
I20260620 18:38:31.745365 28310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:31.728935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.3:58873 (local address 127.23.115.62:36185)
0620 18:38:31.729125 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:31.729130 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:31.730376 (+ 1246us) server_negotiation.cc:407] Connection header received
0620 18:38:31.732181 (+ 1805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:31.732185 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:31.732248 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:31.732327 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:31.733786 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:31.734562 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:31.736117 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:31.736530 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:31.736631 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:31.739826 (+ 3195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:31.739831 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:31.739835 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:31.739869 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:31.742487 (+ 2618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:31.743281 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:31.743287 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:31.743291 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:31.743436 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:31.743849 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:31.743852 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:31.743854 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:31.744169 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:31.744278 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:31.744938 (+ 660us) server_negotiation.cc:299] Negotiation successful
0620 18:38:31.745128 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 18:38:31.746291 28575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:31.729468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:36185 (local address 127.23.115.3:58873)
0620 18:38:31.730216 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:31.730267 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:31.731935 (+ 1668us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:31.732593 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:31.732601 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:31.733105 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:31.733641 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:31.733659 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:31.734704 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:31.734708 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:31.735987 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:31.736000 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:31.736752 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:31.736758 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:31.736944 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:31.737563 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:31.737589 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:31.739616 (+ 2027us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:31.742629 (+ 3013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:31.742638 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:31.742657 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:31.743163 (+ 506us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:31.743546 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:31.743551 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:31.743554 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:31.743736 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:31.744275 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:31.744281 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:31.744610 (+ 329us) client_negotiation.cc:769] Sending connection context
0620 18:38:31.744843 (+ 233us) client_negotiation.cc:240] Negotiation successful
0620 18:38:31.745223 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":312,"threads_started":1}
I20260620 18:38:31.747534 28573 heartbeater.cc:347] Connected to a master server at 127.23.115.62:36185
I20260620 18:38:31.747790 28573 heartbeater.cc:464] Registering TS with master...
I20260620 18:38:31.748396 28573 heartbeater.cc:511] Master 127.23.115.62:36185 requested a full tablet report, sending...
I20260620 18:38:31.749956 28120 ts_manager.cc:194] Registered new tserver with Master: 415c28c7203044549a4da89e04a99fdc (127.23.115.3:36497)
I20260620 18:38:31.750785 28120 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.115.3@KRBTEST.COM'} at 127.23.115.3:58873
I20260620 18:38:31.756033 24012 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:38:31.773373 28310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:31.760549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56704 (local address 127.23.115.62:36185)
0620 18:38:31.760721 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:31.760727 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:31.761232 (+ 505us) server_negotiation.cc:407] Connection header received
0620 18:38:31.761390 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:31.761394 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:31.761455 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:31.761543 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:31.762900 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:31.763665 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:31.764872 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:31.765310 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:31.765417 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:31.767558 (+ 2141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:31.767563 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:31.767566 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:31.767596 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:31.770517 (+ 2921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:31.771232 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:31.771238 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:31.771242 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:31.771374 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:31.771989 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:31.771994 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:31.771997 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:31.772315 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:31.772550 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:31.772856 (+ 306us) server_negotiation.cc:299] Negotiation successful
0620 18:38:31.773065 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 18:38:31.777884 28120 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56704:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:38:31.786731 28120 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:38:31.808717 28308 heartbeater.cc:503] Master 127.23.115.62:36185 was elected leader, sending a full tablet report...
I20260620 18:38:31.823587 28590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:31.799769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52858 (local address 127.23.115.1:37551)
0620 18:38:31.814651 (+ 14882us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:31.814660 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:31.815642 (+ 982us) server_negotiation.cc:407] Connection header received
0620 18:38:31.815736 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:31.815742 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:31.815950 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:31.816119 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:38:31.816952 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:31.818525 (+ 1573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:31.820546 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:31.822105 (+ 1559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:31.822237 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:31.822459 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:31.823076 (+ 617us) server_negotiation.cc:299] Negotiation successful
0620 18:38:31.823232 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14703,"thread_start_us":14541,"threads_started":1}
I20260620 18:38:31.823617 28586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:31.799125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.1:37551 (local address 127.0.0.1:52858)
0620 18:38:31.815405 (+ 16280us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:31.815421 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:31.815521 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:31.816336 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:31.816340 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:31.816361 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:38:31.816760 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:31.816769 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:31.818693 (+ 1924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:31.818700 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:31.820398 (+ 1698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:31.820410 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:31.822624 (+ 2214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:31.822631 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:31.822825 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:31.822876 (+ 51us) client_negotiation.cc:769] Sending connection context
0620 18:38:31.823025 (+ 149us) client_negotiation.cc:240] Negotiation successful
0620 18:38:31.823233 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16148,"mutex_wait_us":6,"thread_start_us":15683,"threads_started":1}
I20260620 18:38:31.823724 28587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:31.798557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34672 (local address 127.23.115.2:46717)
0620 18:38:31.813280 (+ 14723us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:31.813289 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:31.814066 (+ 777us) server_negotiation.cc:407] Connection header received
0620 18:38:31.814672 (+ 606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:31.814683 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:31.815006 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:31.815199 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:38:31.816836 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:31.818189 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:31.820950 (+ 2761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:31.822394 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:31.822531 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:31.822715 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:31.823180 (+ 465us) server_negotiation.cc:299] Negotiation successful
0620 18:38:31.823360 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13416,"thread_start_us":13316,"threads_started":1}
I20260620 18:38:31.823938 28589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:31.799131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55486 (local address 127.23.115.3:36497)
0620 18:38:31.811492 (+ 12361us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:31.811503 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:31.815211 (+ 3708us) server_negotiation.cc:407] Connection header received
0620 18:38:31.815542 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:31.815550 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:31.815759 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:31.815903 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:38:31.816941 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:31.818509 (+ 1568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:31.821100 (+ 2591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:31.822505 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:31.822654 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:31.822891 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:31.823441 (+ 550us) server_negotiation.cc:299] Negotiation successful
0620 18:38:31.823598 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12134,"thread_start_us":12372,"threads_started":1}
I20260620 18:38:31.824699 28585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:31.798332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.2:46717 (local address 127.0.0.1:34672)
0620 18:38:31.813915 (+ 15583us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:31.813961 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:31.814227 (+ 266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:31.815538 (+ 1311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:31.815545 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:31.815578 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:38:31.816020 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:31.816041 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:31.818326 (+ 2285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:31.818332 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:31.820782 (+ 2450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:31.820805 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:31.822664 (+ 1859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:31.822672 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:31.822869 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:31.822905 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 18:38:31.823028 (+ 123us) client_negotiation.cc:240] Negotiation successful
0620 18:38:31.824169 (+ 1141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15295,"mutex_wait_us":958,"thread_start_us":15135,"threads_started":1}
I20260620 18:38:31.824723 28588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:31.798483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.3:36497 (local address 127.0.0.1:55486)
0620 18:38:31.815087 (+ 16604us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:31.815105 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:31.815325 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:31.816202 (+ 877us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:31.816207 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:31.816234 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:38:31.816720 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:31.816732 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:31.818647 (+ 1915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:31.818652 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:31.820959 (+ 2307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:31.820970 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:31.823059 (+ 2089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:31.823066 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:31.823226 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:31.823259 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 18:38:31.824128 (+ 869us) client_negotiation.cc:240] Negotiation successful
0620 18:38:31.824265 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16478,"mutex_wait_us":71,"thread_start_us":16676,"threads_started":1}
I20260620 18:38:31.827569 28243 tablet_service.cc:1467] Processing CreateTablet for tablet 930996414eea4b4db3e77db8dd2a96ec (DEFAULT_TABLE table=test-table [id=91062297cc5d46a18022a98ac2b91474]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:38:31.827992 28376 tablet_service.cc:1467] Processing CreateTablet for tablet 930996414eea4b4db3e77db8dd2a96ec (DEFAULT_TABLE table=test-table [id=91062297cc5d46a18022a98ac2b91474]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:38:31.827992 28508 tablet_service.cc:1467] Processing CreateTablet for tablet 930996414eea4b4db3e77db8dd2a96ec (DEFAULT_TABLE table=test-table [id=91062297cc5d46a18022a98ac2b91474]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:38:31.829015 28243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 930996414eea4b4db3e77db8dd2a96ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:31.829216 28508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 930996414eea4b4db3e77db8dd2a96ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:31.829216 28376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 930996414eea4b4db3e77db8dd2a96ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:31.840523 28591 tablet_bootstrap.cc:492] T 930996414eea4b4db3e77db8dd2a96ec P 9aa158f5757f44b6a00beb2c61417931: Bootstrap starting.
I20260620 18:38:31.843017 28592 tablet_bootstrap.cc:492] T 930996414eea4b4db3e77db8dd2a96ec P 2b6057f1a3ed4e7d939965756ea3c654: Bootstrap starting.
I20260620 18:38:31.844151 28593 tablet_bootstrap.cc:492] T 930996414eea4b4db3e77db8dd2a96ec P 415c28c7203044549a4da89e04a99fdc: Bootstrap starting.
I20260620 18:38:31.845561 28591 tablet_bootstrap.cc:654] T 930996414eea4b4db3e77db8dd2a96ec P 9aa158f5757f44b6a00beb2c61417931: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:31.846637 28592 tablet_bootstrap.cc:654] T 930996414eea4b4db3e77db8dd2a96ec P 2b6057f1a3ed4e7d939965756ea3c654: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:31.846791 28593 tablet_bootstrap.cc:654] T 930996414eea4b4db3e77db8dd2a96ec P 415c28c7203044549a4da89e04a99fdc: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:31.846961 28591 log.cc:824] T 930996414eea4b4db3e77db8dd2a96ec P 9aa158f5757f44b6a00beb2c61417931: Log is configured to *not* fsync() on all Append() calls
I20260620 18:38:31.847882 28592 log.cc:824] T 930996414eea4b4db3e77db8dd2a96ec P 2b6057f1a3ed4e7d939965756ea3c654: Log is configured to *not* fsync() on all Append() calls
I20260620 18:38:31.847853 28593 log.cc:824] T 930996414eea4b4db3e77db8dd2a96ec P 415c28c7203044549a4da89e04a99fdc: Log is configured to *not* fsync() on all Append() calls
I20260620 18:38:31.850390 28591 tablet_bootstrap.cc:492] T 930996414eea4b4db3e77db8dd2a96ec P 9aa158f5757f44b6a00beb2c61417931: No bootstrap required, opened a new log
I20260620 18:38:31.850721 28591 ts_tablet_manager.cc:1397] T 930996414eea4b4db3e77db8dd2a96ec P 9aa158f5757f44b6a00beb2c61417931: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260620 18:38:31.852695 28592 tablet_bootstrap.cc:492] T 930996414eea4b4db3e77db8dd2a96ec P 2b6057f1a3ed4e7d939965756ea3c654: No bootstrap required, opened a new log
I20260620 18:38:31.852931 28592 ts_tablet_manager.cc:1397] T 930996414eea4b4db3e77db8dd2a96ec P 2b6057f1a3ed4e7d939965756ea3c654: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260620 18:38:31.853046 28593 tablet_bootstrap.cc:492] T 930996414eea4b4db3e77db8dd2a96ec P 415c28c7203044549a4da89e04a99fdc: No bootstrap required, opened a new log
I20260620 18:38:31.853335 28593 ts_tablet_manager.cc:1397] T 930996414eea4b4db3e77db8dd2a96ec P 415c28c7203044549a4da89e04a99fdc: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260620 18:38:31.861143 28592 raft_consensus.cc:348] T 930996414eea4b4db3e77db8dd2a96ec P 2b6057f1a3ed4e7d939965756ea3c654 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b6057f1a3ed4e7d939965756ea3c654" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 46717 } } peers { permanent_uuid: "415c28c7203044549a4da89e04a99fdc" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 36497 } } peers { permanent_uuid: "9aa158f5757f44b6a00beb2c61417931" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 37551 } }
I20260620 18:38:31.861692 28592 raft_consensus.cc:374] T 930996414eea4b4db3e77db8dd2a96ec P 2b6057f1a3ed4e7d939965756ea3c654 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:31.862015 28592 raft_consensus.cc:729] T 930996414eea4b4db3e77db8dd2a96ec P 2b6057f1a3ed4e7d939965756ea3c654 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b6057f1a3ed4e7d939965756ea3c654, State: Initialized, Role: FOLLOWER
I20260620 18:38:31.862006 28591 raft_consensus.cc:348] T 930996414eea4b4db3e77db8dd2a96ec P 9aa158f5757f44b6a00beb2c61417931 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b6057f1a3ed4e7d939965756ea3c654" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 46717 } } peers { permanent_uuid: "415c28c7203044549a4da89e04a99fdc" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 36497 } } peers { permanent_uuid: "9aa158f5757f44b6a00beb2c61417931" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 37551 } }
I20260620 18:38:31.862599 28591 raft_consensus.cc:374] T 930996414eea4b4db3e77db8dd2a96ec P 9aa158f5757f44b6a00beb2c61417931 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:31.862717 28591 raft_consensus.cc:729] T 930996414eea4b4db3e77db8dd2a96ec P 9aa158f5757f44b6a00beb2c61417931 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9aa158f5757f44b6a00beb2c61417931, State: Initialized, Role: FOLLOWER
I20260620 18:38:31.862740 28592 consensus_queue.cc:260] T 930996414eea4b4db3e77db8dd2a96ec P 2b6057f1a3ed4e7d939965756ea3c654 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b6057f1a3ed4e7d939965756ea3c654" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 46717 } } peers { permanent_uuid: "415c28c7203044549a4da89e04a99fdc" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 36497 } } peers { permanent_uuid: "9aa158f5757f44b6a00beb2c61417931" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 37551 } }
I20260620 18:38:31.863327 28591 consensus_queue.cc:260] T 930996414eea4b4db3e77db8dd2a96ec P 9aa158f5757f44b6a00beb2c61417931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b6057f1a3ed4e7d939965756ea3c654" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 46717 } } peers { permanent_uuid: "415c28c7203044549a4da89e04a99fdc" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 36497 } } peers { permanent_uuid: "9aa158f5757f44b6a00beb2c61417931" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 37551 } }
I20260620 18:38:31.864044 28441 heartbeater.cc:503] Master 127.23.115.62:36185 was elected leader, sending a full tablet report...
I20260620 18:38:31.864648 28592 ts_tablet_manager.cc:1428] T 930996414eea4b4db3e77db8dd2a96ec P 2b6057f1a3ed4e7d939965756ea3c654: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260620 18:38:31.864616 28593 raft_consensus.cc:348] T 930996414eea4b4db3e77db8dd2a96ec P 415c28c7203044549a4da89e04a99fdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b6057f1a3ed4e7d939965756ea3c654" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 46717 } } peers { permanent_uuid: "415c28c7203044549a4da89e04a99fdc" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 36497 } } peers { permanent_uuid: "9aa158f5757f44b6a00beb2c61417931" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 37551 } }
I20260620 18:38:31.865149 28593 raft_consensus.cc:374] T 930996414eea4b4db3e77db8dd2a96ec P 415c28c7203044549a4da89e04a99fdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:31.865285 28593 raft_consensus.cc:729] T 930996414eea4b4db3e77db8dd2a96ec P 415c28c7203044549a4da89e04a99fdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 415c28c7203044549a4da89e04a99fdc, State: Initialized, Role: FOLLOWER
I20260620 18:38:31.865926 28593 consensus_queue.cc:260] T 930996414eea4b4db3e77db8dd2a96ec P 415c28c7203044549a4da89e04a99fdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b6057f1a3ed4e7d939965756ea3c654" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 46717 } } peers { permanent_uuid: "415c28c7203044549a4da89e04a99fdc" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 36497 } } peers { permanent_uuid: "9aa158f5757f44b6a00beb2c61417931" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 37551 } }
I20260620 18:38:31.867249 28573 heartbeater.cc:503] Master 127.23.115.62:36185 was elected leader, sending a full tablet report...
I20260620 18:38:31.867743 28593 ts_tablet_manager.cc:1428] T 930996414eea4b4db3e77db8dd2a96ec P 415c28c7203044549a4da89e04a99fdc: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260620 18:38:31.870302 28591 ts_tablet_manager.cc:1428] T 930996414eea4b4db3e77db8dd2a96ec P 9aa158f5757f44b6a00beb2c61417931: Time spent starting tablet: real 0.019s user 0.015s sys 0.000s
W20260620 18:38:31.978042 28574 tablet.cc:2369] T 930996414eea4b4db3e77db8dd2a96ec P 415c28c7203044549a4da89e04a99fdc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:38:32.006112 28442 tablet.cc:2369] T 930996414eea4b4db3e77db8dd2a96ec P 2b6057f1a3ed4e7d939965756ea3c654: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:38:32.031322 28309 tablet.cc:2369] T 930996414eea4b4db3e77db8dd2a96ec P 9aa158f5757f44b6a00beb2c61417931: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:38:32.223587 28598 raft_consensus.cc:482] T 930996414eea4b4db3e77db8dd2a96ec P 9aa158f5757f44b6a00beb2c61417931 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:38:32.223872 28598 raft_consensus.cc:504] T 930996414eea4b4db3e77db8dd2a96ec P 9aa158f5757f44b6a00beb2c61417931 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b6057f1a3ed4e7d939965756ea3c654" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 46717 } } peers { permanent_uuid: "415c28c7203044549a4da89e04a99fdc" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 36497 } } peers { permanent_uuid: "9aa158f5757f44b6a00beb2c61417931" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 37551 } }
I20260620 18:38:32.225492 28598 leader_election.cc:288] T 930996414eea4b4db3e77db8dd2a96ec P 9aa158f5757f44b6a00beb2c61417931 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2b6057f1a3ed4e7d939965756ea3c654 (127.23.115.2:46717), 415c28c7203044549a4da89e04a99fdc (127.23.115.3:36497)
I20260620 18:38:32.232267 28589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:32.226058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.1:46417 (local address 127.23.115.3:36497)
0620 18:38:32.226257 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:32.226263 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:32.226843 (+ 580us) server_negotiation.cc:407] Connection header received
0620 18:38:32.227170 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:32.227176 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:32.227251 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:32.227332 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:38:32.227799 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:32.228749 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:32.230622 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:32.231427 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:32.231569 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:32.231643 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:32.231896 (+ 253us) server_negotiation.cc:299] Negotiation successful
0620 18:38:32.232015 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 18:38:32.232349 28600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:32.225745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.3:36497 (local address 127.23.115.1:46417)
0620 18:38:32.226733 (+ 988us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:32.226752 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:32.227006 (+ 254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:32.227360 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:32.227363 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:32.227386 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:38:32.227676 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:32.227685 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:32.228915 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:32.228920 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:32.230476 (+ 1556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:32.230488 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:32.231625 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:32.231631 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:32.231760 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:32.231783 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 18:38:32.231863 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 18:38:32.231984 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":859,"mutex_wait_us":2,"thread_start_us":684,"threads_started":1}
I20260620 18:38:32.233426 28528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "930996414eea4b4db3e77db8dd2a96ec" candidate_uuid: "9aa158f5757f44b6a00beb2c61417931" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "415c28c7203044549a4da89e04a99fdc" is_pre_election: true
I20260620 18:38:32.233675 28587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:32.226099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.1:33813 (local address 127.23.115.2:46717)
0620 18:38:32.226272 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:32.226278 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:32.227186 (+ 908us) server_negotiation.cc:407] Connection header received
0620 18:38:32.227337 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:32.227342 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:32.227412 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:32.227489 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:38:32.228166 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:32.229134 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:32.231607 (+ 2473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:32.232782 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:32.232938 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:32.233005 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:32.233308 (+ 303us) server_negotiation.cc:299] Negotiation successful
0620 18:38:32.233435 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 18:38:32.233737 28601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:32.225747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.2:46717 (local address 127.23.115.1:33813)
0620 18:38:32.227078 (+ 1331us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:32.227101 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:32.227224 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:32.227653 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:32.227661 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:32.227684 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:38:32.228013 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:32.228022 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:32.229280 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:32.229286 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:32.231425 (+ 2139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:32.231439 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:32.232974 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:32.232981 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:32.233149 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:32.233181 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 18:38:32.233266 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 18:38:32.233378 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1218,"mutex_wait_us":552,"thread_start_us":465,"threads_started":1}
I20260620 18:38:32.233922 28528 raft_consensus.cc:2454] T 930996414eea4b4db3e77db8dd2a96ec P 415c28c7203044549a4da89e04a99fdc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9aa158f5757f44b6a00beb2c61417931 in term 0.
I20260620 18:38:32.234592 28198 leader_election.cc:302] T 930996414eea4b4db3e77db8dd2a96ec P 9aa158f5757f44b6a00beb2c61417931 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 415c28c7203044549a4da89e04a99fdc, 9aa158f5757f44b6a00beb2c61417931; no voters:
I20260620 18:38:32.234606 28396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "930996414eea4b4db3e77db8dd2a96ec" candidate_uuid: "9aa158f5757f44b6a00beb2c61417931" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b6057f1a3ed4e7d939965756ea3c654" is_pre_election: true
I20260620 18:38:32.235023 28598 raft_consensus.cc:2781] T 930996414eea4b4db3e77db8dd2a96ec P 9aa158f5757f44b6a00beb2c61417931 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:38:32.235034 28396 raft_consensus.cc:2454] T 930996414eea4b4db3e77db8dd2a96ec P 2b6057f1a3ed4e7d939965756ea3c654 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9aa158f5757f44b6a00beb2c61417931 in term 0.
I20260620 18:38:32.235133 28598 raft_consensus.cc:482] T 930996414eea4b4db3e77db8dd2a96ec P 9aa158f5757f44b6a00beb2c61417931 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:38:32.235229 28598 raft_consensus.cc:3037] T 930996414eea4b4db3e77db8dd2a96ec P 9aa158f5757f44b6a00beb2c61417931 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:32.237355 28598 raft_consensus.cc:504] T 930996414eea4b4db3e77db8dd2a96ec P 9aa158f5757f44b6a00beb2c61417931 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b6057f1a3ed4e7d939965756ea3c654" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 46717 } } peers { permanent_uuid: "415c28c7203044549a4da89e04a99fdc" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 36497 } } peers { permanent_uuid: "9aa158f5757f44b6a00beb2c61417931" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 37551 } }
I20260620 18:38:32.237928 28598 leader_election.cc:288] T 930996414eea4b4db3e77db8dd2a96ec P 9aa158f5757f44b6a00beb2c61417931 [CANDIDATE]: Term 1 election: Requested vote from peers 2b6057f1a3ed4e7d939965756ea3c654 (127.23.115.2:46717), 415c28c7203044549a4da89e04a99fdc (127.23.115.3:36497)
I20260620 18:38:32.238340 28396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "930996414eea4b4db3e77db8dd2a96ec" candidate_uuid: "9aa158f5757f44b6a00beb2c61417931" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b6057f1a3ed4e7d939965756ea3c654"
I20260620 18:38:32.238399 28528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "930996414eea4b4db3e77db8dd2a96ec" candidate_uuid: "9aa158f5757f44b6a00beb2c61417931" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "415c28c7203044549a4da89e04a99fdc"
I20260620 18:38:32.238503 28396 raft_consensus.cc:3037] T 930996414eea4b4db3e77db8dd2a96ec P 2b6057f1a3ed4e7d939965756ea3c654 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:32.238600 28528 raft_consensus.cc:3037] T 930996414eea4b4db3e77db8dd2a96ec P 415c28c7203044549a4da89e04a99fdc [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:32.240628 28396 raft_consensus.cc:2454] T 930996414eea4b4db3e77db8dd2a96ec P 2b6057f1a3ed4e7d939965756ea3c654 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9aa158f5757f44b6a00beb2c61417931 in term 1.
I20260620 18:38:32.240806 28528 raft_consensus.cc:2454] T 930996414eea4b4db3e77db8dd2a96ec P 415c28c7203044549a4da89e04a99fdc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9aa158f5757f44b6a00beb2c61417931 in term 1.
I20260620 18:38:32.241034 28197 leader_election.cc:302] T 930996414eea4b4db3e77db8dd2a96ec P 9aa158f5757f44b6a00beb2c61417931 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b6057f1a3ed4e7d939965756ea3c654, 9aa158f5757f44b6a00beb2c61417931; no voters:
I20260620 18:38:32.241331 28598 raft_consensus.cc:2781] T 930996414eea4b4db3e77db8dd2a96ec P 9aa158f5757f44b6a00beb2c61417931 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:38:32.241729 28598 raft_consensus.cc:686] T 930996414eea4b4db3e77db8dd2a96ec P 9aa158f5757f44b6a00beb2c61417931 [term 1 LEADER]: Becoming Leader. State: Replica: 9aa158f5757f44b6a00beb2c61417931, State: Running, Role: LEADER
I20260620 18:38:32.242177 28598 consensus_queue.cc:237] T 930996414eea4b4db3e77db8dd2a96ec P 9aa158f5757f44b6a00beb2c61417931 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b6057f1a3ed4e7d939965756ea3c654" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 46717 } } peers { permanent_uuid: "415c28c7203044549a4da89e04a99fdc" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 36497 } } peers { permanent_uuid: "9aa158f5757f44b6a00beb2c61417931" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 37551 } }
I20260620 18:38:32.246624 28120 catalog_manager.cc:5495] T 930996414eea4b4db3e77db8dd2a96ec P 9aa158f5757f44b6a00beb2c61417931 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9aa158f5757f44b6a00beb2c61417931 (127.23.115.1). New cstate: current_term: 1 leader_uuid: "9aa158f5757f44b6a00beb2c61417931" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b6057f1a3ed4e7d939965756ea3c654" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 46717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "415c28c7203044549a4da89e04a99fdc" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 36497 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9aa158f5757f44b6a00beb2c61417931" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 37551 } health_report { overall_health: HEALTHY } } }
I20260620 18:38:32.271090 28590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:32.265499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52872 (local address 127.23.115.1:37551)
0620 18:38:32.265676 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:32.265681 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:32.265715 (+ 34us) server_negotiation.cc:407] Connection header received
0620 18:38:32.265881 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:32.265884 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:32.265958 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:32.266075 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:38:32.266422 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:32.267163 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:32.268425 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:32.269117 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:32.269228 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:32.269751 (+ 523us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:38:32.270251 (+ 500us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:38:32.270371 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:32.270740 (+ 369us) server_negotiation.cc:299] Negotiation successful
0620 18:38:32.270818 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 18:38:32.279206 28396 raft_consensus.cc:1261] T 930996414eea4b4db3e77db8dd2a96ec P 2b6057f1a3ed4e7d939965756ea3c654 [term 1 FOLLOWER]: Refusing update from remote peer 9aa158f5757f44b6a00beb2c61417931: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:38:32.279206 28528 raft_consensus.cc:1261] T 930996414eea4b4db3e77db8dd2a96ec P 415c28c7203044549a4da89e04a99fdc [term 1 FOLLOWER]: Refusing update from remote peer 9aa158f5757f44b6a00beb2c61417931: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:38:32.280340 28602 consensus_queue.cc:1035] T 930996414eea4b4db3e77db8dd2a96ec P 9aa158f5757f44b6a00beb2c61417931 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b6057f1a3ed4e7d939965756ea3c654" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 46717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:38:32.280680 28598 consensus_queue.cc:1035] T 930996414eea4b4db3e77db8dd2a96ec P 9aa158f5757f44b6a00beb2c61417931 [LEADER]: Connected to new peer: Peer: permanent_uuid: "415c28c7203044549a4da89e04a99fdc" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 36497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:38:32.303390 28119 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56704:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:38:32.303656 28119 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56704:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:38:32.307265 28119 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b9c52c861bc34190bc131f82443df04a: Sending DeleteTablet for 3 replicas of tablet 930996414eea4b4db3e77db8dd2a96ec
I20260620 18:38:32.308394 28376 tablet_service.cc:1514] Processing DeleteTablet for tablet 930996414eea4b4db3e77db8dd2a96ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:38:32 UTC) from {username='kudu', principal='kudu/127.23.115.62@KRBTEST.COM'} at 127.0.0.1:34672
I20260620 18:38:32.308672 28243 tablet_service.cc:1514] Processing DeleteTablet for tablet 930996414eea4b4db3e77db8dd2a96ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:38:32 UTC) from {username='kudu', principal='kudu/127.23.115.62@KRBTEST.COM'} at 127.0.0.1:52858
I20260620 18:38:32.309079 28508 tablet_service.cc:1514] Processing DeleteTablet for tablet 930996414eea4b4db3e77db8dd2a96ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:38:32 UTC) from {username='kudu', principal='kudu/127.23.115.62@KRBTEST.COM'} at 127.0.0.1:55486
I20260620 18:38:32.310298 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 28181
I20260620 18:38:32.331899 28617 tablet_replica.cc:330] T 930996414eea4b4db3e77db8dd2a96ec P 415c28c7203044549a4da89e04a99fdc: stopping tablet replica
I20260620 18:38:32.332525 28617 raft_consensus.cc:2229] T 930996414eea4b4db3e77db8dd2a96ec P 415c28c7203044549a4da89e04a99fdc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:38:32.333189 28617 raft_consensus.cc:2258] T 930996414eea4b4db3e77db8dd2a96ec P 415c28c7203044549a4da89e04a99fdc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:38:32.333804 28615 tablet_replica.cc:330] T 930996414eea4b4db3e77db8dd2a96ec P 2b6057f1a3ed4e7d939965756ea3c654: stopping tablet replica
I20260620 18:38:32.334618 28615 raft_consensus.cc:2229] T 930996414eea4b4db3e77db8dd2a96ec P 2b6057f1a3ed4e7d939965756ea3c654 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:38:32.335168 28615 raft_consensus.cc:2258] T 930996414eea4b4db3e77db8dd2a96ec P 2b6057f1a3ed4e7d939965756ea3c654 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:38:32.335942 28616 tablet_replica.cc:330] T 930996414eea4b4db3e77db8dd2a96ec P 9aa158f5757f44b6a00beb2c61417931: stopping tablet replica
I20260620 18:38:32.336324 28616 raft_consensus.cc:2229] T 930996414eea4b4db3e77db8dd2a96ec P 9aa158f5757f44b6a00beb2c61417931 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:38:32.336843 28616 raft_consensus.cc:2258] T 930996414eea4b4db3e77db8dd2a96ec P 9aa158f5757f44b6a00beb2c61417931 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:38:32.337354 28615 ts_tablet_manager.cc:1905] T 930996414eea4b4db3e77db8dd2a96ec P 2b6057f1a3ed4e7d939965756ea3c654: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:38:32.337354 28617 ts_tablet_manager.cc:1905] T 930996414eea4b4db3e77db8dd2a96ec P 415c28c7203044549a4da89e04a99fdc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:38:32.338843 28616 ts_tablet_manager.cc:1905] T 930996414eea4b4db3e77db8dd2a96ec P 9aa158f5757f44b6a00beb2c61417931: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:38:32.344375 28615 ts_tablet_manager.cc:1918] T 930996414eea4b4db3e77db8dd2a96ec P 2b6057f1a3ed4e7d939965756ea3c654: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:38:32.344631 28615 log.cc:1196] T 930996414eea4b4db3e77db8dd2a96ec P 2b6057f1a3ed4e7d939965756ea3c654: Deleting WAL directory at /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980676253068-24012-0/minicluster-data/ts-1/wal/wals/930996414eea4b4db3e77db8dd2a96ec
I20260620 18:38:32.345261 28615 ts_tablet_manager.cc:1939] T 930996414eea4b4db3e77db8dd2a96ec P 2b6057f1a3ed4e7d939965756ea3c654: Deleting consensus metadata
I20260620 18:38:32.346602 28616 ts_tablet_manager.cc:1918] T 930996414eea4b4db3e77db8dd2a96ec P 9aa158f5757f44b6a00beb2c61417931: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:38:32.346834 28616 log.cc:1196] T 930996414eea4b4db3e77db8dd2a96ec P 9aa158f5757f44b6a00beb2c61417931: Deleting WAL directory at /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980676253068-24012-0/minicluster-data/ts-0/wal/wals/930996414eea4b4db3e77db8dd2a96ec
I20260620 18:38:32.347458 28616 ts_tablet_manager.cc:1939] T 930996414eea4b4db3e77db8dd2a96ec P 9aa158f5757f44b6a00beb2c61417931: Deleting consensus metadata
I20260620 18:38:32.347673 28105 catalog_manager.cc:4841] TS 2b6057f1a3ed4e7d939965756ea3c654 (127.23.115.2:46717): tablet 930996414eea4b4db3e77db8dd2a96ec (table test-table [id=91062297cc5d46a18022a98ac2b91474]) successfully deleted
I20260620 18:38:32.349107 28104 catalog_manager.cc:4841] TS 9aa158f5757f44b6a00beb2c61417931 (127.23.115.1:37551): tablet 930996414eea4b4db3e77db8dd2a96ec (table test-table [id=91062297cc5d46a18022a98ac2b91474]) successfully deleted
I20260620 18:38:32.349414 28617 ts_tablet_manager.cc:1918] T 930996414eea4b4db3e77db8dd2a96ec P 415c28c7203044549a4da89e04a99fdc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:38:32.349603 28617 log.cc:1196] T 930996414eea4b4db3e77db8dd2a96ec P 415c28c7203044549a4da89e04a99fdc: Deleting WAL directory at /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980676253068-24012-0/minicluster-data/ts-2/wal/wals/930996414eea4b4db3e77db8dd2a96ec
I20260620 18:38:32.350136 28617 ts_tablet_manager.cc:1939] T 930996414eea4b4db3e77db8dd2a96ec P 415c28c7203044549a4da89e04a99fdc: Deleting consensus metadata
Jun 20 18:38:32 dist-test-slave-d645 krb5kdc[28076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980709, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.115.1@KRBTEST.COM
I20260620 18:38:32.351847 28106 catalog_manager.cc:4841] TS 415c28c7203044549a4da89e04a99fdc (127.23.115.3:36497): tablet 930996414eea4b4db3e77db8dd2a96ec (table test-table [id=91062297cc5d46a18022a98ac2b91474]) successfully deleted
I20260620 18:38:32.357973 28590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:32.312431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52878 (local address 127.23.115.1:37551)
0620 18:38:32.312624 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:32.312631 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:32.331476 (+ 18845us) server_negotiation.cc:407] Connection header received
0620 18:38:32.331954 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:32.331965 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:32.332072 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:32.332153 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:32.334050 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:32.335261 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:32.344810 (+ 9549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:32.345559 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:32.345676 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:32.351374 (+ 5698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:32.351389 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:32.351411 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:32.351464 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:32.355148 (+ 3684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:32.355858 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:32.355867 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:32.355871 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:32.356041 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:32.356371 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:32.356376 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:32.356379 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:32.356873 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:32.357154 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:32.357353 (+ 199us) server_negotiation.cc:299] Negotiation successful
0620 18:38:32.357592 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 18:38:32.358752 28303 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:38:32.456276 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 28181
I20260620 18:38:32.471347 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 28314
Jun 20 18:38:32 dist-test-slave-d645 krb5kdc[28076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980709, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.115.2@KRBTEST.COM
I20260620 18:38:32.485708 28587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:32.472146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34678 (local address 127.23.115.2:46717)
0620 18:38:32.472348 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:32.472354 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:32.472373 (+ 19us) server_negotiation.cc:407] Connection header received
0620 18:38:32.472579 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:32.472585 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:32.472668 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:32.472754 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:32.473934 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:32.474647 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:32.475920 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:32.476361 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:32.476475 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:32.480535 (+ 4060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:32.480556 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:32.480573 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:32.480616 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:32.483215 (+ 2599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:32.483856 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:32.483863 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:32.483865 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:32.484002 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:32.484382 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:32.484386 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:32.484388 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:32.484784 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:32.484998 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:32.485289 (+ 291us) server_negotiation.cc:299] Negotiation successful
0620 18:38:32.485450 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 18:38:32.486294 28436 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:38:32.572185 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 28314
I20260620 18:38:32.585559 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 28446
Jun 20 18:38:32 dist-test-slave-d645 krb5kdc[28076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980709, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.115.3@KRBTEST.COM
I20260620 18:38:32.601387 28589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:32.586286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55488 (local address 127.23.115.3:36497)
0620 18:38:32.586445 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:32.586456 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:32.586876 (+ 420us) server_negotiation.cc:407] Connection header received
0620 18:38:32.587109 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:32.587113 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:32.587193 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:32.587309 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:32.588747 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:32.589445 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:32.590733 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:32.591231 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:32.591346 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:32.595762 (+ 4416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:32.595775 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:32.595793 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:32.595831 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:32.598581 (+ 2750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:32.599292 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:32.599300 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:32.599302 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:32.599440 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:32.599922 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:32.599926 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:32.599928 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:32.600279 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:32.600550 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:32.600861 (+ 311us) server_negotiation.cc:299] Negotiation successful
0620 18:38:32.601065 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 18:38:32.601996 28568 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:38:32.687616 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 28446
I20260620 18:38:32.700475 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 28089
I20260620 18:38:32.701606 28150 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:38:32.811893 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 28089
2026-06-20T18:38:32Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4191 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-20T18:38:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:38:32Z Disabled control of system clock
I20260620 18:38:32.871706 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980676253068-24012-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980676253068-24012-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980676253068-24012-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.115.62:46613
--webserver_interface=127.23.115.62
--webserver_port=0
--builtin_ntp_servers=127.23.115.20:40211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.115.62:46613
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 18:38:33.199872 28626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:33.200265 28626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:33.200376 28626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:33.209142 28626 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:38:33.209286 28626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:33.209340 28626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:33.209431 28626 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:38:33.209496 28626 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:38:33.221813 28626 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:40211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980676253068-24012-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980676253068-24012-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.115.62:46613
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.115.62:46613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.115.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980676253068-24012-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:38:33.223629 28626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:33.225725 28626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:33.239254 28631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:33.239229 28634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:33.239209 28632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:33.239898 28626 server_base.cc:1034] running on GCE node
I20260620 18:38:33.240669 28626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:33.242057 28626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:33.243285 28626 hybrid_clock.cc:648] HybridClock initialized: now 1781980713243228 us; error 47 us; skew 500 ppm
I20260620 18:38:33.243649 28626 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:38:33.245962 28626 webserver.cc:460] Webserver started at http://127.23.115.62:39877/ using document root <none> and password file <none>
I20260620 18:38:33.246666 28626 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:33.246765 28626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:33.247107 28626 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:38:33.249899 28626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980676253068-24012-0/minicluster-data/master-0/data/instance:
uuid: "1f1ea5d35d4544788b99d5057a055541"
format_stamp: "Formatted at 2026-06-20 18:38:33 on dist-test-slave-d645"
server_key: "42c64fbdfc8ff0bb2bd0d0f03fb6c420"
server_key_iv: "60a28471130f651a9aed0899c58e4dad"
server_key_version: "encryptionkey@0"
I20260620 18:38:33.250692 28626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980676253068-24012-0/minicluster-data/master-0/wal/instance:
uuid: "1f1ea5d35d4544788b99d5057a055541"
format_stamp: "Formatted at 2026-06-20 18:38:33 on dist-test-slave-d645"
server_key: "42c64fbdfc8ff0bb2bd0d0f03fb6c420"
server_key_iv: "60a28471130f651a9aed0899c58e4dad"
server_key_version: "encryptionkey@0"
I20260620 18:38:33.255959 28626 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 18:38:33.260088 28640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:33.261569 28626 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 18:38:33.261734 28626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980676253068-24012-0/minicluster-data/master-0/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980676253068-24012-0/minicluster-data/master-0/wal
uuid: "1f1ea5d35d4544788b99d5057a055541"
format_stamp: "Formatted at 2026-06-20 18:38:33 on dist-test-slave-d645"
server_key: "42c64fbdfc8ff0bb2bd0d0f03fb6c420"
server_key_iv: "60a28471130f651a9aed0899c58e4dad"
server_key_version: "encryptionkey@0"
I20260620 18:38:33.261986 28626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980676253068-24012-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980676253068-24012-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980676253068-24012-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:33.267683 28626 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:33.268419 28626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:33.268687 28626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:33.286180 28626 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.62:46613
I20260620 18:38:33.286221 28691 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.62:46613 every 8 connection(s)
I20260620 18:38:33.287477 28626 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
I20260620 18:38:33.291121 28692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:33.291492 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 28626
I20260620 18:38:33.291692 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980676253068-24012-0/minicluster-data/master-0/wal/instance
I20260620 18:38:33.292486 24012 external_mini_cluster.cc:1428] Setting key 68ec6597d6a5da9101fafada159cee0a
I20260620 18:38:33.302053 28695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:33.295432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48216 (local address 127.23.115.62:46613)
0620 18:38:33.296588 (+ 1156us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:33.296602 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:33.296649 (+ 47us) server_negotiation.cc:407] Connection header received
0620 18:38:33.298152 (+ 1503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:33.298174 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:33.298832 (+ 658us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:33.299292 (+ 460us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:33.300423 (+ 1131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:33.300429 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:33.300445 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:38:33.300462 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:33.300526 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:38:33.300548 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:33.300737 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:33.300832 (+ 95us) server_negotiation.cc:299] Negotiation successful
0620 18:38:33.301075 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":708,"thread_start_us":470,"threads_started":1}
I20260620 18:38:33.305145 28692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f1ea5d35d4544788b99d5057a055541: Bootstrap starting.
I20260620 18:38:33.308030 28692 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1f1ea5d35d4544788b99d5057a055541: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:33.308983 28692 log.cc:824] T 00000000000000000000000000000000 P 1f1ea5d35d4544788b99d5057a055541: Log is configured to *not* fsync() on all Append() calls
I20260620 18:38:33.311506 28692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f1ea5d35d4544788b99d5057a055541: No bootstrap required, opened a new log
I20260620 18:38:33.318444 28692 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1f1ea5d35d4544788b99d5057a055541 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f1ea5d35d4544788b99d5057a055541" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 46613 } }
I20260620 18:38:33.318787 28692 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1f1ea5d35d4544788b99d5057a055541 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:33.318877 28692 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1f1ea5d35d4544788b99d5057a055541 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f1ea5d35d4544788b99d5057a055541, State: Initialized, Role: FOLLOWER
I20260620 18:38:33.319499 28692 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1f1ea5d35d4544788b99d5057a055541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f1ea5d35d4544788b99d5057a055541" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 46613 } }
I20260620 18:38:33.319725 28692 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1f1ea5d35d4544788b99d5057a055541 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:38:33.319844 28692 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1f1ea5d35d4544788b99d5057a055541 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:38:33.319983 28692 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1f1ea5d35d4544788b99d5057a055541 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:33.322443 28692 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1f1ea5d35d4544788b99d5057a055541 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f1ea5d35d4544788b99d5057a055541" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 46613 } }
I20260620 18:38:33.322919 28692 leader_election.cc:302] T 00000000000000000000000000000000 P 1f1ea5d35d4544788b99d5057a055541 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1f1ea5d35d4544788b99d5057a055541; no voters:
I20260620 18:38:33.323486 28692 leader_election.cc:288] T 00000000000000000000000000000000 P 1f1ea5d35d4544788b99d5057a055541 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:38:33.323643 28697 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1f1ea5d35d4544788b99d5057a055541 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:38:33.324313 28697 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1f1ea5d35d4544788b99d5057a055541 [term 1 LEADER]: Becoming Leader. State: Replica: 1f1ea5d35d4544788b99d5057a055541, State: Running, Role: LEADER
I20260620 18:38:33.324810 28697 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1f1ea5d35d4544788b99d5057a055541 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f1ea5d35d4544788b99d5057a055541" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 46613 } }
I20260620 18:38:33.325451 28692 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1f1ea5d35d4544788b99d5057a055541 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:38:33.327706 28699 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f1ea5d35d4544788b99d5057a055541 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1f1ea5d35d4544788b99d5057a055541. Latest consensus state: current_term: 1 leader_uuid: "1f1ea5d35d4544788b99d5057a055541" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f1ea5d35d4544788b99d5057a055541" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 46613 } } }
I20260620 18:38:33.328151 28699 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f1ea5d35d4544788b99d5057a055541 [sys.catalog]: This master's current role is: LEADER
I20260620 18:38:33.328094 28698 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f1ea5d35d4544788b99d5057a055541 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1f1ea5d35d4544788b99d5057a055541" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f1ea5d35d4544788b99d5057a055541" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 46613 } } }
I20260620 18:38:33.328411 28698 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f1ea5d35d4544788b99d5057a055541 [sys.catalog]: This master's current role is: LEADER
I20260620 18:38:33.330130 28706 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:38:33.335348 28706 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:38:33.344596 28706 catalog_manager.cc:1347] Generated new cluster ID: 65049ebe58c54a64b20782792203ca1f
I20260620 18:38:33.344769 28706 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:38:33.365125 28706 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:38:33.365595 28706 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:38:33.381656 28706 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1f1ea5d35d4544788b99d5057a055541: Generated new TSK 0
I20260620 18:38:33.382550 28706 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:38:33.388831 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980676253068-24012-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980676253068-24012-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980676253068-24012-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980676253068-24012-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.115.1:0
--local_ip_for_outbound_sockets=127.23.115.1
--webserver_interface=127.23.115.1
--webserver_port=0
--tserver_master_addrs=127.23.115.62:46613
--builtin_ntp_servers=127.23.115.20:40211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 18:38:33.711814 28716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:33.712086 28716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:33.712200 28716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:33.719393 28716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:33.719493 28716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:33.719628 28716 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.115.1
I20260620 18:38:33.728907 28716 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:40211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980676253068-24012-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980676253068-24012-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.115.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980676253068-24012-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.115.1
--webserver_port=0
--tserver_master_addrs=127.23.115.62:46613
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.115.1
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980676253068-24012-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:38:33.730202 28716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:33.732409 28716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:33.745584 28721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:33.745657 28722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:33.746161 28724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:33.746450 28716 server_base.cc:1034] running on GCE node
I20260620 18:38:33.747114 28716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:33.748108 28716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:33.749347 28716 hybrid_clock.cc:648] HybridClock initialized: now 1781980713749292 us; error 47 us; skew 500 ppm
I20260620 18:38:33.749713 28716 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:38:33.752115 28716 webserver.cc:460] Webserver started at http://127.23.115.1:46059/ using document root <none> and password file <none>
I20260620 18:38:33.752799 28716 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:33.752897 28716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:33.753306 28716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:38:33.756721 28716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980676253068-24012-0/minicluster-data/ts-0/data/instance:
uuid: "25e8ffa05df14841b468d5e664bd3c3c"
format_stamp: "Formatted at 2026-06-20 18:38:33 on dist-test-slave-d645"
server_key: "8e3fa70fa9423166dd4eade5609d12ac"
server_key_iv: "9d08dbba7c8dc817f266880fdae30fdd"
server_key_version: "encryptionkey@0"
I20260620 18:38:33.757524 28716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980676253068-24012-0/minicluster-data/ts-0/wal/instance:
uuid: "25e8ffa05df14841b468d5e664bd3c3c"
format_stamp: "Formatted at 2026-06-20 18:38:33 on dist-test-slave-d645"
server_key: "8e3fa70fa9423166dd4eade5609d12ac"
server_key_iv: "9d08dbba7c8dc817f266880fdae30fdd"
server_key_version: "encryptionkey@0"
I20260620 18:38:33.762715 28716 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 18:38:33.766319 28730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:33.767665 28716 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:38:33.767825 28716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980676253068-24012-0/minicluster-data/ts-0/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980676253068-24012-0/minicluster-data/ts-0/wal
uuid: "25e8ffa05df14841b468d5e664bd3c3c"
format_stamp: "Formatted at 2026-06-20 18:38:33 on dist-test-slave-d645"
server_key: "8e3fa70fa9423166dd4eade5609d12ac"
server_key_iv: "9d08dbba7c8dc817f266880fdae30fdd"
server_key_version: "encryptionkey@0"
I20260620 18:38:33.768035 28716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980676253068-24012-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980676253068-24012-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980676253068-24012-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:33.774643 28716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:33.775579 28716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:33.775853 28716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:33.776859 28716 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:38:33.778376 28716 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:38:33.778483 28716 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:33.778597 28716 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:38:33.778676 28716 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:33.811108 28716 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.1:36667
I20260620 18:38:33.811179 28842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.1:36667 every 8 connection(s)
I20260620 18:38:33.812362 28716 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980676253068-24012-0/minicluster-data/ts-0/data/info.pb
I20260620 18:38:33.817389 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 28716
I20260620 18:38:33.817627 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980676253068-24012-0/minicluster-data/ts-0/wal/instance
I20260620 18:38:33.818387 24012 external_mini_cluster.cc:1428] Setting key a4158d2583681b4cf76487cf4ab73886
I20260620 18:38:33.821553 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980676253068-24012-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980676253068-24012-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980676253068-24012-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980676253068-24012-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.115.2:0
--local_ip_for_outbound_sockets=127.23.115.2
--webserver_interface=127.23.115.2
--webserver_port=0
--tserver_master_addrs=127.23.115.62:46613
--builtin_ntp_servers=127.23.115.20:40211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260620 18:38:33.822324 28845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:33.815834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.1:43117 (local address 127.23.115.62:46613)
0620 18:38:33.816473 (+ 639us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:33.816482 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:33.817601 (+ 1119us) server_negotiation.cc:407] Connection header received
0620 18:38:33.819761 (+ 2160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:33.819767 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:33.819868 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:33.820011 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:33.821460 (+ 1449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:33.821464 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:33.821468 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:38:33.821484 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:33.821526 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:38:33.821538 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:33.821702 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:33.822024 (+ 322us) server_negotiation.cc:299] Negotiation successful
0620 18:38:33.822104 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":302,"threads_started":1}
I20260620 18:38:33.823659 28846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:33.816315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:46613 (local address 127.23.115.1:43117)
0620 18:38:33.817419 (+ 1104us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:33.817475 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:33.819506 (+ 2031us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:33.820257 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:33.820270 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:33.820402 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:33.821101 (+ 699us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:38:33.821105 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:33.821156 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:38:33.821270 (+ 114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:33.821725 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:33.821737 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:33.821748 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 18:38:33.822095 (+ 347us) client_negotiation.cc:240] Negotiation successful
0620 18:38:33.822370 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":323,"threads_started":1}
I20260620 18:38:33.825145 28843 heartbeater.cc:347] Connected to a master server at 127.23.115.62:46613
I20260620 18:38:33.825421 28843 heartbeater.cc:464] Registering TS with master...
I20260620 18:38:33.826056 28843 heartbeater.cc:511] Master 127.23.115.62:46613 requested a full tablet report, sending...
I20260620 18:38:33.828507 28657 ts_manager.cc:194] Registered new tserver with Master: 25e8ffa05df14841b468d5e664bd3c3c (127.23.115.1:36667)
W20260620 18:38:34.149405 28847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:34.149694 28847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:34.149822 28847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:34.157050 28847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:34.157174 28847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:34.157320 28847 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.115.2
I20260620 18:38:34.166607 28847 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:40211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980676253068-24012-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980676253068-24012-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.115.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980676253068-24012-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.115.2
--webserver_port=0
--tserver_master_addrs=127.23.115.62:46613
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.115.2
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980676253068-24012-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:38:34.168102 28847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:34.169991 28847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:34.187275 28852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:34.187338 28853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:34.187361 28855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:34.188465 28847 server_base.cc:1034] running on GCE node
I20260620 18:38:34.189146 28847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:34.190119 28847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:34.191325 28847 hybrid_clock.cc:648] HybridClock initialized: now 1781980714191264 us; error 51 us; skew 500 ppm
I20260620 18:38:34.191653 28847 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:38:34.194262 28847 webserver.cc:460] Webserver started at http://127.23.115.2:39103/ using document root <none> and password file <none>
I20260620 18:38:34.194970 28847 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:34.195078 28847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:34.195399 28847 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:38:34.198256 28847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980676253068-24012-0/minicluster-data/ts-1/data/instance:
uuid: "509467a2d4d642f984044373e7ee91dc"
format_stamp: "Formatted at 2026-06-20 18:38:34 on dist-test-slave-d645"
server_key: "bcc9bdfc5481f68ccbbfb764fbbbe1eb"
server_key_iv: "218df6a36d165225a6ffd1055a96d406"
server_key_version: "encryptionkey@0"
I20260620 18:38:34.199112 28847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980676253068-24012-0/minicluster-data/ts-1/wal/instance:
uuid: "509467a2d4d642f984044373e7ee91dc"
format_stamp: "Formatted at 2026-06-20 18:38:34 on dist-test-slave-d645"
server_key: "bcc9bdfc5481f68ccbbfb764fbbbe1eb"
server_key_iv: "218df6a36d165225a6ffd1055a96d406"
server_key_version: "encryptionkey@0"
I20260620 18:38:34.204463 28847 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 18:38:34.208415 28861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:34.210122 28847 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260620 18:38:34.210309 28847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980676253068-24012-0/minicluster-data/ts-1/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980676253068-24012-0/minicluster-data/ts-1/wal
uuid: "509467a2d4d642f984044373e7ee91dc"
format_stamp: "Formatted at 2026-06-20 18:38:34 on dist-test-slave-d645"
server_key: "bcc9bdfc5481f68ccbbfb764fbbbe1eb"
server_key_iv: "218df6a36d165225a6ffd1055a96d406"
server_key_version: "encryptionkey@0"
I20260620 18:38:34.210521 28847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980676253068-24012-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980676253068-24012-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980676253068-24012-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:34.216238 28847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:34.217026 28847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:34.217273 28847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:34.218279 28847 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:38:34.219672 28847 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:38:34.219766 28847 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:34.219875 28847 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:38:34.219941 28847 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:34.253083 28847 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.2:41675
I20260620 18:38:34.253106 28973 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.2:41675 every 8 connection(s)
I20260620 18:38:34.254462 28847 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980676253068-24012-0/minicluster-data/ts-1/data/info.pb
I20260620 18:38:34.262640 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 28847
I20260620 18:38:34.262809 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980676253068-24012-0/minicluster-data/ts-1/wal/instance
I20260620 18:38:34.263482 24012 external_mini_cluster.cc:1428] Setting key 96e397d67eabdca6e1959d4ed191cbc1
I20260620 18:38:34.267434 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980676253068-24012-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980676253068-24012-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980676253068-24012-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980676253068-24012-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.115.3:0
--local_ip_for_outbound_sockets=127.23.115.3
--webserver_interface=127.23.115.3
--webserver_port=0
--tserver_master_addrs=127.23.115.62:46613
--builtin_ntp_servers=127.23.115.20:40211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260620 18:38:34.267690 28845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:34.257244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.2:53331 (local address 127.23.115.62:46613)
0620 18:38:34.257438 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:34.257442 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:34.258846 (+ 1404us) server_negotiation.cc:407] Connection header received
0620 18:38:34.260724 (+ 1878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:34.260731 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:34.260823 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:34.260927 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:34.266839 (+ 5912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:34.266844 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:34.266848 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:38:34.266864 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:34.266896 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:38:34.266908 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:34.267040 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:34.267328 (+ 288us) server_negotiation.cc:299] Negotiation successful
0620 18:38:34.267395 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 18:38:34.268740 28976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:34.257654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:46613 (local address 127.23.115.2:53331)
0620 18:38:34.258652 (+ 998us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:34.258705 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:34.260428 (+ 1723us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:34.261180 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:34.261188 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:34.261284 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:34.266454 (+ 5170us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:38:34.266457 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:34.266538 (+ 81us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:38:34.266650 (+ 112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:34.267101 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:34.267112 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:34.267121 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 18:38:34.267411 (+ 290us) client_negotiation.cc:240] Negotiation successful
0620 18:38:34.267714 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":330,"threads_started":1}
I20260620 18:38:34.270726 28974 heartbeater.cc:347] Connected to a master server at 127.23.115.62:46613
I20260620 18:38:34.271175 28974 heartbeater.cc:464] Registering TS with master...
I20260620 18:38:34.272068 28974 heartbeater.cc:511] Master 127.23.115.62:46613 requested a full tablet report, sending...
I20260620 18:38:34.274232 28657 ts_manager.cc:194] Registered new tserver with Master: 509467a2d4d642f984044373e7ee91dc (127.23.115.2:41675)
W20260620 18:38:34.612604 28977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:34.612921 28977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:34.613046 28977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:34.620580 28977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:34.620709 28977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:34.620846 28977 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.115.3
I20260620 18:38:34.630534 28977 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:40211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980676253068-24012-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980676253068-24012-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.115.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980676253068-24012-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.115.3
--webserver_port=0
--tserver_master_addrs=127.23.115.62:46613
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.115.3
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980676253068-24012-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:38:34.631843 28977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:34.633575 28977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:34.646464 28983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:34.646517 28982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:34.646612 28985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:34.647627 28977 server_base.cc:1034] running on GCE node
I20260620 18:38:34.648335 28977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:34.649294 28977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:34.650528 28977 hybrid_clock.cc:648] HybridClock initialized: now 1781980714650465 us; error 52 us; skew 500 ppm
I20260620 18:38:34.650862 28977 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:38:34.653117 28977 webserver.cc:460] Webserver started at http://127.23.115.3:36521/ using document root <none> and password file <none>
I20260620 18:38:34.653757 28977 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:34.653844 28977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:34.654114 28977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:38:34.657685 28977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980676253068-24012-0/minicluster-data/ts-2/data/instance:
uuid: "43095bb6a1d64e7fbab997b2c59c2501"
format_stamp: "Formatted at 2026-06-20 18:38:34 on dist-test-slave-d645"
server_key: "4f3ea648a2e1fc13f640173fa3766d21"
server_key_iv: "8fb53cfc2c6cf53c2bcecc03c8a075fa"
server_key_version: "encryptionkey@0"
I20260620 18:38:34.658483 28977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980676253068-24012-0/minicluster-data/ts-2/wal/instance:
uuid: "43095bb6a1d64e7fbab997b2c59c2501"
format_stamp: "Formatted at 2026-06-20 18:38:34 on dist-test-slave-d645"
server_key: "4f3ea648a2e1fc13f640173fa3766d21"
server_key_iv: "8fb53cfc2c6cf53c2bcecc03c8a075fa"
server_key_version: "encryptionkey@0"
I20260620 18:38:34.663553 28977 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 18:38:34.667126 28991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:34.668506 28977 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:38:34.668658 28977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980676253068-24012-0/minicluster-data/ts-2/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980676253068-24012-0/minicluster-data/ts-2/wal
uuid: "43095bb6a1d64e7fbab997b2c59c2501"
format_stamp: "Formatted at 2026-06-20 18:38:34 on dist-test-slave-d645"
server_key: "4f3ea648a2e1fc13f640173fa3766d21"
server_key_iv: "8fb53cfc2c6cf53c2bcecc03c8a075fa"
server_key_version: "encryptionkey@0"
I20260620 18:38:34.668840 28977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980676253068-24012-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980676253068-24012-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980676253068-24012-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:34.674364 28977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:34.675174 28977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:34.675390 28977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:34.676222 28977 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:38:34.677553 28977 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:38:34.677634 28977 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:34.677713 28977 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:38:34.677767 28977 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:34.717595 28977 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.3:40909
I20260620 18:38:34.717641 29103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.3:40909 every 8 connection(s)
I20260620 18:38:34.718861 28977 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980676253068-24012-0/minicluster-data/ts-2/data/info.pb
I20260620 18:38:34.719620 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 28977
I20260620 18:38:34.719791 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980676253068-24012-0/minicluster-data/ts-2/wal/instance
I20260620 18:38:34.720323 24012 external_mini_cluster.cc:1428] Setting key 65148c6288cbd639dc6a3d15895c470b
I20260620 18:38:34.727346 28845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:34.722320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.3:36607 (local address 127.23.115.62:46613)
0620 18:38:34.722521 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:34.722527 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:34.723007 (+ 480us) server_negotiation.cc:407] Connection header received
0620 18:38:34.725067 (+ 2060us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:34.725073 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:34.725148 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:34.725250 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:34.726620 (+ 1370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:34.726626 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:34.726632 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:38:34.726651 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:34.726681 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:38:34.726692 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:34.726801 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:34.727011 (+ 210us) server_negotiation.cc:299] Negotiation successful
0620 18:38:34.727076 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 18:38:34.728266 29106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:34.721849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:46613 (local address 127.23.115.3:36607)
0620 18:38:34.722856 (+ 1007us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:34.722904 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:34.724774 (+ 1870us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:34.725504 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:34.725513 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:34.725611 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:34.726278 (+ 667us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:38:34.726282 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:34.726336 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:38:34.726444 (+ 108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:34.726779 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:34.726789 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:34.726797 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 18:38:34.727085 (+ 288us) client_negotiation.cc:240] Negotiation successful
0620 18:38:34.727376 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":300,"threads_started":1}
I20260620 18:38:34.729628 29104 heartbeater.cc:347] Connected to a master server at 127.23.115.62:46613
I20260620 18:38:34.729935 29104 heartbeater.cc:464] Registering TS with master...
I20260620 18:38:34.730561 29104 heartbeater.cc:511] Master 127.23.115.62:46613 requested a full tablet report, sending...
I20260620 18:38:34.732360 28657 ts_manager.cc:194] Registered new tserver with Master: 43095bb6a1d64e7fbab997b2c59c2501 (127.23.115.3:40909)
I20260620 18:38:34.737124 24012 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:38:34.745029 28845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:34.742537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48230 (local address 127.23.115.62:46613)
0620 18:38:34.742786 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:34.742790 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:34.743264 (+ 474us) server_negotiation.cc:407] Connection header received
0620 18:38:34.743389 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:34.743393 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:34.743452 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:34.743567 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:34.744355 (+ 788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:34.744359 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:34.744363 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:38:34.744378 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:34.744406 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:38:34.744417 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:34.744516 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:34.744726 (+ 210us) server_negotiation.cc:299] Negotiation successful
0620 18:38:34.744779 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260620 18:38:34.750485 28657 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48230:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:38:34.757863 28657 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:38:34.780366 29118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:34.771222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.2:41675 (local address 127.0.0.1:39420)
0620 18:38:34.772975 (+ 1753us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:34.773018 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:34.773162 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:34.775537 (+ 2375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:34.775546 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:34.775654 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:34.776463 (+ 809us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:38:34.776467 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:34.776504 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:38:34.776591 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:34.779603 (+ 3012us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:34.779616 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:34.779626 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 18:38:34.779809 (+ 183us) client_negotiation.cc:240] Negotiation successful
0620 18:38:34.779906 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1465,"threads_started":1}
I20260620 18:38:34.781353 29117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:34.771430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39420 (local address 127.23.115.2:41675)
0620 18:38:34.773670 (+ 2240us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:34.773680 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:34.773710 (+ 30us) server_negotiation.cc:407] Connection header received
0620 18:38:34.773777 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:34.773784 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:34.774038 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:34.774190 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:34.777679 (+ 3489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:34.777689 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:34.777706 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:38:34.777734 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:34.777789 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:38:34.777808 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:34.780813 (+ 3005us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:34.780911 (+ 98us) server_negotiation.cc:299] Negotiation successful
0620 18:38:34.781026 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2085,"thread_start_us":2183,"threads_started":1}
I20260620 18:38:34.786883 29118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:34.771486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.3:40909 (local address 127.0.0.1:50086)
0620 18:38:34.780598 (+ 9112us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:34.780618 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:34.780698 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:34.782695 (+ 1997us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:34.782701 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:34.782729 (+ 28us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:34.784741 (+ 2012us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:38:34.784745 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:34.784766 (+ 21us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:38:34.784827 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:34.786395 (+ 1568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:34.786401 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:34.786407 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 18:38:34.786497 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 18:38:34.786555 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9001,"thread_start_us":1000,"threads_started":1}
I20260620 18:38:34.787827 29119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:34.772061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50086 (local address 127.23.115.3:40909)
0620 18:38:34.782050 (+ 9989us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:34.782059 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:34.782099 (+ 40us) server_negotiation.cc:407] Connection header received
0620 18:38:34.782172 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:34.782181 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:34.782452 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:34.784591 (+ 2139us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:34.786121 (+ 1530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:34.786131 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:34.786145 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:38:34.786165 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:34.786230 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:38:34.786252 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:34.787237 (+ 985us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:34.787328 (+ 91us) server_negotiation.cc:299] Negotiation successful
0620 18:38:34.787433 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9815,"thread_start_us":9654,"threads_started":1}
I20260620 18:38:34.788813 29116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:34.774015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.1:36667 (local address 127.0.0.1:55930)
0620 18:38:34.785228 (+ 11213us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:34.785247 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:34.785369 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:34.786902 (+ 1533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:34.786908 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:34.786960 (+ 52us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:34.787673 (+ 713us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:38:34.787678 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:34.787697 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:38:34.787751 (+ 54us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:34.788226 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:34.788231 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:34.788236 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 18:38:34.788374 (+ 138us) client_negotiation.cc:240] Negotiation successful
0620 18:38:34.788424 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11056,"thread_start_us":13574,"threads_started":1}
I20260620 18:38:34.789839 29121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:34.774198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55930 (local address 127.23.115.1:36667)
0620 18:38:34.784522 (+ 10324us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:34.784531 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:34.785500 (+ 969us) server_negotiation.cc:407] Connection header received
0620 18:38:34.785586 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:34.785596 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:34.785830 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:34.786038 (+ 208us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:34.787948 (+ 1910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:34.787958 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:34.787971 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:38:34.788002 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:34.788066 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:38:34.788089 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:34.788230 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:34.788390 (+ 160us) server_negotiation.cc:299] Negotiation successful
0620 18:38:34.788484 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10153,"thread_start_us":10012,"threads_started":1}
I20260620 18:38:34.789342 28909 tablet_service.cc:1467] Processing CreateTablet for tablet b437fa132c6d4594b3d730912698ec96 (DEFAULT_TABLE table=test-table [id=204fd92e4ba147f09457625f09164ea4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:38:34.790795 28909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b437fa132c6d4594b3d730912698ec96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:34.791528 29039 tablet_service.cc:1467] Processing CreateTablet for tablet b437fa132c6d4594b3d730912698ec96 (DEFAULT_TABLE table=test-table [id=204fd92e4ba147f09457625f09164ea4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:38:34.792681 29039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b437fa132c6d4594b3d730912698ec96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:34.793401 28778 tablet_service.cc:1467] Processing CreateTablet for tablet b437fa132c6d4594b3d730912698ec96 (DEFAULT_TABLE table=test-table [id=204fd92e4ba147f09457625f09164ea4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:38:34.794651 28778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b437fa132c6d4594b3d730912698ec96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:34.804630 29122 tablet_bootstrap.cc:492] T b437fa132c6d4594b3d730912698ec96 P 43095bb6a1d64e7fbab997b2c59c2501: Bootstrap starting.
I20260620 18:38:34.806743 29123 tablet_bootstrap.cc:492] T b437fa132c6d4594b3d730912698ec96 P 509467a2d4d642f984044373e7ee91dc: Bootstrap starting.
I20260620 18:38:34.808671 29122 tablet_bootstrap.cc:654] T b437fa132c6d4594b3d730912698ec96 P 43095bb6a1d64e7fbab997b2c59c2501: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:34.808955 29124 tablet_bootstrap.cc:492] T b437fa132c6d4594b3d730912698ec96 P 25e8ffa05df14841b468d5e664bd3c3c: Bootstrap starting.
I20260620 18:38:34.810098 29122 log.cc:824] T b437fa132c6d4594b3d730912698ec96 P 43095bb6a1d64e7fbab997b2c59c2501: Log is configured to *not* fsync() on all Append() calls
I20260620 18:38:34.810259 29123 tablet_bootstrap.cc:654] T b437fa132c6d4594b3d730912698ec96 P 509467a2d4d642f984044373e7ee91dc: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:34.811360 29123 log.cc:824] T b437fa132c6d4594b3d730912698ec96 P 509467a2d4d642f984044373e7ee91dc: Log is configured to *not* fsync() on all Append() calls
I20260620 18:38:34.813371 29122 tablet_bootstrap.cc:492] T b437fa132c6d4594b3d730912698ec96 P 43095bb6a1d64e7fbab997b2c59c2501: No bootstrap required, opened a new log
I20260620 18:38:34.813632 29122 ts_tablet_manager.cc:1397] T b437fa132c6d4594b3d730912698ec96 P 43095bb6a1d64e7fbab997b2c59c2501: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.002s
I20260620 18:38:34.815102 29124 tablet_bootstrap.cc:654] T b437fa132c6d4594b3d730912698ec96 P 25e8ffa05df14841b468d5e664bd3c3c: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:34.815685 29123 tablet_bootstrap.cc:492] T b437fa132c6d4594b3d730912698ec96 P 509467a2d4d642f984044373e7ee91dc: No bootstrap required, opened a new log
I20260620 18:38:34.816016 29123 ts_tablet_manager.cc:1397] T b437fa132c6d4594b3d730912698ec96 P 509467a2d4d642f984044373e7ee91dc: Time spent bootstrapping tablet: real 0.009s user 0.009s sys 0.000s
I20260620 18:38:34.816397 29124 log.cc:824] T b437fa132c6d4594b3d730912698ec96 P 25e8ffa05df14841b468d5e664bd3c3c: Log is configured to *not* fsync() on all Append() calls
I20260620 18:38:34.824357 29122 raft_consensus.cc:348] T b437fa132c6d4594b3d730912698ec96 P 43095bb6a1d64e7fbab997b2c59c2501 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "509467a2d4d642f984044373e7ee91dc" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 41675 } } peers { permanent_uuid: "43095bb6a1d64e7fbab997b2c59c2501" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 40909 } } peers { permanent_uuid: "25e8ffa05df14841b468d5e664bd3c3c" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 36667 } }
I20260620 18:38:34.824867 29122 raft_consensus.cc:374] T b437fa132c6d4594b3d730912698ec96 P 43095bb6a1d64e7fbab997b2c59c2501 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:34.824966 29122 raft_consensus.cc:729] T b437fa132c6d4594b3d730912698ec96 P 43095bb6a1d64e7fbab997b2c59c2501 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43095bb6a1d64e7fbab997b2c59c2501, State: Initialized, Role: FOLLOWER
I20260620 18:38:34.825711 29122 consensus_queue.cc:260] T b437fa132c6d4594b3d730912698ec96 P 43095bb6a1d64e7fbab997b2c59c2501 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "509467a2d4d642f984044373e7ee91dc" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 41675 } } peers { permanent_uuid: "43095bb6a1d64e7fbab997b2c59c2501" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 40909 } } peers { permanent_uuid: "25e8ffa05df14841b468d5e664bd3c3c" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 36667 } }
I20260620 18:38:34.826992 29124 tablet_bootstrap.cc:492] T b437fa132c6d4594b3d730912698ec96 P 25e8ffa05df14841b468d5e664bd3c3c: No bootstrap required, opened a new log
I20260620 18:38:34.827307 29124 ts_tablet_manager.cc:1397] T b437fa132c6d4594b3d730912698ec96 P 25e8ffa05df14841b468d5e664bd3c3c: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.010s
I20260620 18:38:34.827767 29104 heartbeater.cc:503] Master 127.23.115.62:46613 was elected leader, sending a full tablet report...
I20260620 18:38:34.828423 29122 ts_tablet_manager.cc:1428] T b437fa132c6d4594b3d730912698ec96 P 43095bb6a1d64e7fbab997b2c59c2501: Time spent starting tablet: real 0.015s user 0.011s sys 0.004s
I20260620 18:38:34.828137 29123 raft_consensus.cc:348] T b437fa132c6d4594b3d730912698ec96 P 509467a2d4d642f984044373e7ee91dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "509467a2d4d642f984044373e7ee91dc" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 41675 } } peers { permanent_uuid: "43095bb6a1d64e7fbab997b2c59c2501" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 40909 } } peers { permanent_uuid: "25e8ffa05df14841b468d5e664bd3c3c" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 36667 } }
I20260620 18:38:34.828553 29123 raft_consensus.cc:374] T b437fa132c6d4594b3d730912698ec96 P 509467a2d4d642f984044373e7ee91dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:34.828639 29123 raft_consensus.cc:729] T b437fa132c6d4594b3d730912698ec96 P 509467a2d4d642f984044373e7ee91dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 509467a2d4d642f984044373e7ee91dc, State: Initialized, Role: FOLLOWER
I20260620 18:38:34.829339 29123 consensus_queue.cc:260] T b437fa132c6d4594b3d730912698ec96 P 509467a2d4d642f984044373e7ee91dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "509467a2d4d642f984044373e7ee91dc" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 41675 } } peers { permanent_uuid: "43095bb6a1d64e7fbab997b2c59c2501" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 40909 } } peers { permanent_uuid: "25e8ffa05df14841b468d5e664bd3c3c" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 36667 } }
I20260620 18:38:34.832530 28843 heartbeater.cc:503] Master 127.23.115.62:46613 was elected leader, sending a full tablet report...
I20260620 18:38:34.833408 29123 ts_tablet_manager.cc:1428] T b437fa132c6d4594b3d730912698ec96 P 509467a2d4d642f984044373e7ee91dc: Time spent starting tablet: real 0.017s user 0.007s sys 0.008s
I20260620 18:38:34.833446 28974 heartbeater.cc:503] Master 127.23.115.62:46613 was elected leader, sending a full tablet report...
I20260620 18:38:34.838562 29124 raft_consensus.cc:348] T b437fa132c6d4594b3d730912698ec96 P 25e8ffa05df14841b468d5e664bd3c3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "509467a2d4d642f984044373e7ee91dc" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 41675 } } peers { permanent_uuid: "43095bb6a1d64e7fbab997b2c59c2501" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 40909 } } peers { permanent_uuid: "25e8ffa05df14841b468d5e664bd3c3c" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 36667 } }
I20260620 18:38:34.839252 29124 raft_consensus.cc:374] T b437fa132c6d4594b3d730912698ec96 P 25e8ffa05df14841b468d5e664bd3c3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:34.839371 29124 raft_consensus.cc:729] T b437fa132c6d4594b3d730912698ec96 P 25e8ffa05df14841b468d5e664bd3c3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25e8ffa05df14841b468d5e664bd3c3c, State: Initialized, Role: FOLLOWER
I20260620 18:38:34.840006 29124 consensus_queue.cc:260] T b437fa132c6d4594b3d730912698ec96 P 25e8ffa05df14841b468d5e664bd3c3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "509467a2d4d642f984044373e7ee91dc" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 41675 } } peers { permanent_uuid: "43095bb6a1d64e7fbab997b2c59c2501" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 40909 } } peers { permanent_uuid: "25e8ffa05df14841b468d5e664bd3c3c" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 36667 } }
I20260620 18:38:34.841674 29124 ts_tablet_manager.cc:1428] T b437fa132c6d4594b3d730912698ec96 P 25e8ffa05df14841b468d5e664bd3c3c: Time spent starting tablet: real 0.014s user 0.011s sys 0.001s
I20260620 18:38:34.904654 29130 raft_consensus.cc:482] T b437fa132c6d4594b3d730912698ec96 P 25e8ffa05df14841b468d5e664bd3c3c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:38:34.904953 29130 raft_consensus.cc:504] T b437fa132c6d4594b3d730912698ec96 P 25e8ffa05df14841b468d5e664bd3c3c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "509467a2d4d642f984044373e7ee91dc" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 41675 } } peers { permanent_uuid: "43095bb6a1d64e7fbab997b2c59c2501" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 40909 } } peers { permanent_uuid: "25e8ffa05df14841b468d5e664bd3c3c" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 36667 } }
I20260620 18:38:34.906672 29130 leader_election.cc:288] T b437fa132c6d4594b3d730912698ec96 P 25e8ffa05df14841b468d5e664bd3c3c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 509467a2d4d642f984044373e7ee91dc (127.23.115.2:41675), 43095bb6a1d64e7fbab997b2c59c2501 (127.23.115.3:40909)
I20260620 18:38:34.909698 29119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:34.907755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.1:39973 (local address 127.23.115.3:40909)
0620 18:38:34.907928 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:34.907933 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:34.907952 (+ 19us) server_negotiation.cc:407] Connection header received
0620 18:38:34.908011 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:34.908016 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:34.908060 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:34.908147 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:34.909123 (+ 976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:34.909129 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:34.909133 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:38:34.909150 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:34.909179 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:38:34.909190 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:34.909313 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:34.909389 (+ 76us) server_negotiation.cc:299] Negotiation successful
0620 18:38:34.909451 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 18:38:34.909909 29131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:34.906968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.3:40909 (local address 127.23.115.1:39973)
0620 18:38:34.907692 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:34.907716 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:34.907854 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:34.908249 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:34.908254 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:34.908278 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:34.908901 (+ 623us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:38:34.908904 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:34.908922 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:38:34.908971 (+ 49us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:34.909280 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:34.909285 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:34.909289 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 18:38:34.909412 (+ 123us) client_negotiation.cc:240] Negotiation successful
0620 18:38:34.909472 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":326,"threads_started":1}
I20260620 18:38:34.909950 29117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:34.907262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.1:55875 (local address 127.23.115.2:41675)
0620 18:38:34.907455 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:34.907461 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:34.908153 (+ 692us) server_negotiation.cc:407] Connection header received
0620 18:38:34.908301 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:34.908306 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:34.908353 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:34.908432 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:34.909334 (+ 902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:34.909340 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:34.909343 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:38:34.909359 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:34.909389 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:38:34.909399 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:34.909544 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:34.909616 (+ 72us) server_negotiation.cc:299] Negotiation successful
0620 18:38:34.909688 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 18:38:34.910148 29132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:34.907137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.2:41675 (local address 127.23.115.1:55875)
0620 18:38:34.908054 (+ 917us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:34.908069 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:34.908165 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:34.908510 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:34.908519 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:34.908543 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:34.909122 (+ 579us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:38:34.909125 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:34.909138 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:38:34.909173 (+ 35us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:34.909506 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:34.909511 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:34.909515 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 18:38:34.909655 (+ 140us) client_negotiation.cc:240] Negotiation successful
0620 18:38:34.909700 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":800,"thread_start_us":616,"threads_started":1}
I20260620 18:38:34.911144 29059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b437fa132c6d4594b3d730912698ec96" candidate_uuid: "25e8ffa05df14841b468d5e664bd3c3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43095bb6a1d64e7fbab997b2c59c2501" is_pre_election: true
I20260620 18:38:34.911250 28929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b437fa132c6d4594b3d730912698ec96" candidate_uuid: "25e8ffa05df14841b468d5e664bd3c3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "509467a2d4d642f984044373e7ee91dc" is_pre_election: true
I20260620 18:38:34.911646 29059 raft_consensus.cc:2454] T b437fa132c6d4594b3d730912698ec96 P 43095bb6a1d64e7fbab997b2c59c2501 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 25e8ffa05df14841b468d5e664bd3c3c in term 0.
I20260620 18:38:34.911695 28929 raft_consensus.cc:2454] T b437fa132c6d4594b3d730912698ec96 P 509467a2d4d642f984044373e7ee91dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 25e8ffa05df14841b468d5e664bd3c3c in term 0.
I20260620 18:38:34.912304 28732 leader_election.cc:302] T b437fa132c6d4594b3d730912698ec96 P 25e8ffa05df14841b468d5e664bd3c3c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25e8ffa05df14841b468d5e664bd3c3c, 509467a2d4d642f984044373e7ee91dc; no voters:
I20260620 18:38:34.912668 29130 raft_consensus.cc:2781] T b437fa132c6d4594b3d730912698ec96 P 25e8ffa05df14841b468d5e664bd3c3c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:38:34.912812 29130 raft_consensus.cc:482] T b437fa132c6d4594b3d730912698ec96 P 25e8ffa05df14841b468d5e664bd3c3c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:38:34.912922 29130 raft_consensus.cc:3037] T b437fa132c6d4594b3d730912698ec96 P 25e8ffa05df14841b468d5e664bd3c3c [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:34.915110 29130 raft_consensus.cc:504] T b437fa132c6d4594b3d730912698ec96 P 25e8ffa05df14841b468d5e664bd3c3c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "509467a2d4d642f984044373e7ee91dc" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 41675 } } peers { permanent_uuid: "43095bb6a1d64e7fbab997b2c59c2501" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 40909 } } peers { permanent_uuid: "25e8ffa05df14841b468d5e664bd3c3c" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 36667 } }
I20260620 18:38:34.915709 29130 leader_election.cc:288] T b437fa132c6d4594b3d730912698ec96 P 25e8ffa05df14841b468d5e664bd3c3c [CANDIDATE]: Term 1 election: Requested vote from peers 509467a2d4d642f984044373e7ee91dc (127.23.115.2:41675), 43095bb6a1d64e7fbab997b2c59c2501 (127.23.115.3:40909)
I20260620 18:38:34.916222 28929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b437fa132c6d4594b3d730912698ec96" candidate_uuid: "25e8ffa05df14841b468d5e664bd3c3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "509467a2d4d642f984044373e7ee91dc"
I20260620 18:38:34.916257 29059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b437fa132c6d4594b3d730912698ec96" candidate_uuid: "25e8ffa05df14841b468d5e664bd3c3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43095bb6a1d64e7fbab997b2c59c2501"
I20260620 18:38:34.916461 28929 raft_consensus.cc:3037] T b437fa132c6d4594b3d730912698ec96 P 509467a2d4d642f984044373e7ee91dc [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:34.916450 29059 raft_consensus.cc:3037] T b437fa132c6d4594b3d730912698ec96 P 43095bb6a1d64e7fbab997b2c59c2501 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:34.919819 28929 raft_consensus.cc:2454] T b437fa132c6d4594b3d730912698ec96 P 509467a2d4d642f984044373e7ee91dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25e8ffa05df14841b468d5e664bd3c3c in term 1.
I20260620 18:38:34.919819 29059 raft_consensus.cc:2454] T b437fa132c6d4594b3d730912698ec96 P 43095bb6a1d64e7fbab997b2c59c2501 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25e8ffa05df14841b468d5e664bd3c3c in term 1.
I20260620 18:38:34.920331 28731 leader_election.cc:302] T b437fa132c6d4594b3d730912698ec96 P 25e8ffa05df14841b468d5e664bd3c3c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 25e8ffa05df14841b468d5e664bd3c3c, 43095bb6a1d64e7fbab997b2c59c2501, 509467a2d4d642f984044373e7ee91dc; no voters:
I20260620 18:38:34.920818 29130 raft_consensus.cc:2781] T b437fa132c6d4594b3d730912698ec96 P 25e8ffa05df14841b468d5e664bd3c3c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:38:34.921839 29130 raft_consensus.cc:686] T b437fa132c6d4594b3d730912698ec96 P 25e8ffa05df14841b468d5e664bd3c3c [term 1 LEADER]: Becoming Leader. State: Replica: 25e8ffa05df14841b468d5e664bd3c3c, State: Running, Role: LEADER
I20260620 18:38:34.922533 29130 consensus_queue.cc:237] T b437fa132c6d4594b3d730912698ec96 P 25e8ffa05df14841b468d5e664bd3c3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "509467a2d4d642f984044373e7ee91dc" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 41675 } } peers { permanent_uuid: "43095bb6a1d64e7fbab997b2c59c2501" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 40909 } } peers { permanent_uuid: "25e8ffa05df14841b468d5e664bd3c3c" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 36667 } }
I20260620 18:38:34.927539 28657 catalog_manager.cc:5495] T b437fa132c6d4594b3d730912698ec96 P 25e8ffa05df14841b468d5e664bd3c3c reported cstate change: term changed from 0 to 1, leader changed from <none> to 25e8ffa05df14841b468d5e664bd3c3c (127.23.115.1). New cstate: current_term: 1 leader_uuid: "25e8ffa05df14841b468d5e664bd3c3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "509467a2d4d642f984044373e7ee91dc" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 41675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43095bb6a1d64e7fbab997b2c59c2501" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 40909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "25e8ffa05df14841b468d5e664bd3c3c" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 36667 } health_report { overall_health: HEALTHY } } }
W20260620 18:38:34.970628 29105 tablet.cc:2369] T b437fa132c6d4594b3d730912698ec96 P 43095bb6a1d64e7fbab997b2c59c2501: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:38:34.973839 29121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:34.971585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55942 (local address 127.23.115.1:36667)
0620 18:38:34.971771 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:34.971777 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:34.971988 (+ 211us) server_negotiation.cc:407] Connection header received
0620 18:38:34.972155 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:34.972160 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:34.972249 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:34.972354 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:34.973339 (+ 985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:34.973344 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:34.973347 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:38:34.973382 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:34.973413 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:38:34.973425 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:34.973514 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:34.973589 (+ 75us) server_negotiation.cc:299] Negotiation successful
0620 18:38:34.973634 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 18:38:34.982909 28929 raft_consensus.cc:1261] T b437fa132c6d4594b3d730912698ec96 P 509467a2d4d642f984044373e7ee91dc [term 1 FOLLOWER]: Refusing update from remote peer 25e8ffa05df14841b468d5e664bd3c3c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:38:34.982909 29059 raft_consensus.cc:1261] T b437fa132c6d4594b3d730912698ec96 P 43095bb6a1d64e7fbab997b2c59c2501 [term 1 FOLLOWER]: Refusing update from remote peer 25e8ffa05df14841b468d5e664bd3c3c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:38:34.984009 29133 consensus_queue.cc:1035] T b437fa132c6d4594b3d730912698ec96 P 25e8ffa05df14841b468d5e664bd3c3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "43095bb6a1d64e7fbab997b2c59c2501" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 40909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:38:34.984407 29130 consensus_queue.cc:1035] T b437fa132c6d4594b3d730912698ec96 P 25e8ffa05df14841b468d5e664bd3c3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "509467a2d4d642f984044373e7ee91dc" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 41675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:38:34.999392 29136 mvcc.cc:205] Tried to move back new op lower bound from 7298993008557232128 to 7298993008327528448. Current Snapshot: MvccSnapshot[applied={T|T < 7298993008557232128}]
I20260620 18:38:35.024283 28657 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48230:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:38:35.024643 28657 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48230:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:38:35.028963 28657 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1f1ea5d35d4544788b99d5057a055541: Sending DeleteTablet for 3 replicas of tablet b437fa132c6d4594b3d730912698ec96
I20260620 18:38:35.030607 28909 tablet_service.cc:1514] Processing DeleteTablet for tablet b437fa132c6d4594b3d730912698ec96 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:38:35 UTC) from {username='slave'} at 127.0.0.1:39420
I20260620 18:38:35.030663 28778 tablet_service.cc:1514] Processing DeleteTablet for tablet b437fa132c6d4594b3d730912698ec96 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:38:35 UTC) from {username='slave'} at 127.0.0.1:55930
I20260620 18:38:35.031275 29039 tablet_service.cc:1514] Processing DeleteTablet for tablet b437fa132c6d4594b3d730912698ec96 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:38:35 UTC) from {username='slave'} at 127.0.0.1:50086
I20260620 18:38:35.033533 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 28716
I20260620 18:38:35.045903 29146 tablet_replica.cc:330] T b437fa132c6d4594b3d730912698ec96 P 509467a2d4d642f984044373e7ee91dc: stopping tablet replica
I20260620 18:38:35.046456 29146 raft_consensus.cc:2229] T b437fa132c6d4594b3d730912698ec96 P 509467a2d4d642f984044373e7ee91dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:38:35.047076 29146 raft_consensus.cc:2258] T b437fa132c6d4594b3d730912698ec96 P 509467a2d4d642f984044373e7ee91dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:38:35.047919 29147 tablet_replica.cc:330] T b437fa132c6d4594b3d730912698ec96 P 25e8ffa05df14841b468d5e664bd3c3c: stopping tablet replica
I20260620 18:38:35.048302 29147 raft_consensus.cc:2229] T b437fa132c6d4594b3d730912698ec96 P 25e8ffa05df14841b468d5e664bd3c3c [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:38:35.048926 29147 raft_consensus.cc:2258] T b437fa132c6d4594b3d730912698ec96 P 25e8ffa05df14841b468d5e664bd3c3c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:38:35.050673 29147 ts_tablet_manager.cc:1905] T b437fa132c6d4594b3d730912698ec96 P 25e8ffa05df14841b468d5e664bd3c3c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:38:35.054859 29148 tablet_replica.cc:330] T b437fa132c6d4594b3d730912698ec96 P 43095bb6a1d64e7fbab997b2c59c2501: stopping tablet replica
I20260620 18:38:35.055424 29148 raft_consensus.cc:2229] T b437fa132c6d4594b3d730912698ec96 P 43095bb6a1d64e7fbab997b2c59c2501 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:38:35.056025 29148 raft_consensus.cc:2258] T b437fa132c6d4594b3d730912698ec96 P 43095bb6a1d64e7fbab997b2c59c2501 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:38:35.056923 29147 ts_tablet_manager.cc:1918] T b437fa132c6d4594b3d730912698ec96 P 25e8ffa05df14841b468d5e664bd3c3c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:38:35.057013 29121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:35.034872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55946 (local address 127.23.115.1:36667)
0620 18:38:35.035088 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:35.035094 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:35.054203 (+ 19109us) server_negotiation.cc:407] Connection header received
0620 18:38:35.054445 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:35.054453 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:35.054534 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:35.054691 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:35.056083 (+ 1392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:35.056090 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:35.056096 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:38:35.056118 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:35.056157 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:38:35.056175 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:35.056293 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:35.056604 (+ 311us) server_negotiation.cc:299] Negotiation successful
0620 18:38:35.056676 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260620 18:38:35.057116 29147 log.cc:1196] T b437fa132c6d4594b3d730912698ec96 P 25e8ffa05df14841b468d5e664bd3c3c: Deleting WAL directory at /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980676253068-24012-0/minicluster-data/ts-0/wal/wals/b437fa132c6d4594b3d730912698ec96
I20260620 18:38:35.057674 29147 ts_tablet_manager.cc:1939] T b437fa132c6d4594b3d730912698ec96 P 25e8ffa05df14841b468d5e664bd3c3c: Deleting consensus metadata
I20260620 18:38:35.058290 28838 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:38:35.059711 29146 ts_tablet_manager.cc:1905] T b437fa132c6d4594b3d730912698ec96 P 509467a2d4d642f984044373e7ee91dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:38:35.059756 29148 ts_tablet_manager.cc:1905] T b437fa132c6d4594b3d730912698ec96 P 43095bb6a1d64e7fbab997b2c59c2501: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:38:35.067412 29148 ts_tablet_manager.cc:1918] T b437fa132c6d4594b3d730912698ec96 P 43095bb6a1d64e7fbab997b2c59c2501: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:38:35.067394 29146 ts_tablet_manager.cc:1918] T b437fa132c6d4594b3d730912698ec96 P 509467a2d4d642f984044373e7ee91dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:38:35.067606 29148 log.cc:1196] T b437fa132c6d4594b3d730912698ec96 P 43095bb6a1d64e7fbab997b2c59c2501: Deleting WAL directory at /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980676253068-24012-0/minicluster-data/ts-2/wal/wals/b437fa132c6d4594b3d730912698ec96
I20260620 18:38:35.067589 29146 log.cc:1196] T b437fa132c6d4594b3d730912698ec96 P 509467a2d4d642f984044373e7ee91dc: Deleting WAL directory at /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980676253068-24012-0/minicluster-data/ts-1/wal/wals/b437fa132c6d4594b3d730912698ec96
I20260620 18:38:35.068114 29146 ts_tablet_manager.cc:1939] T b437fa132c6d4594b3d730912698ec96 P 509467a2d4d642f984044373e7ee91dc: Deleting consensus metadata
I20260620 18:38:35.068114 29148 ts_tablet_manager.cc:1939] T b437fa132c6d4594b3d730912698ec96 P 43095bb6a1d64e7fbab997b2c59c2501: Deleting consensus metadata
I20260620 18:38:35.070034 28642 catalog_manager.cc:4841] TS 509467a2d4d642f984044373e7ee91dc (127.23.115.2:41675): tablet b437fa132c6d4594b3d730912698ec96 (table test-table [id=204fd92e4ba147f09457625f09164ea4]) successfully deleted
I20260620 18:38:35.070014 28641 catalog_manager.cc:4841] TS 43095bb6a1d64e7fbab997b2c59c2501 (127.23.115.3:40909): tablet b437fa132c6d4594b3d730912698ec96 (table test-table [id=204fd92e4ba147f09457625f09164ea4]) successfully deleted
I20260620 18:38:35.164615 28643 catalog_manager.cc:4841] TS 25e8ffa05df14841b468d5e664bd3c3c (127.23.115.1:36667): tablet b437fa132c6d4594b3d730912698ec96 (table test-table [id=204fd92e4ba147f09457625f09164ea4]) successfully deleted
I20260620 18:38:35.164988 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 28716
I20260620 18:38:35.180374 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 28847
I20260620 18:38:35.183763 29117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:35.181393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39430 (local address 127.23.115.2:41675)
0620 18:38:35.181566 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:35.181570 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:35.181592 (+ 22us) server_negotiation.cc:407] Connection header received
0620 18:38:35.181732 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:35.181737 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:35.181826 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:35.181921 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:35.183099 (+ 1178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:35.183104 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:35.183107 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:38:35.183123 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:35.183158 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:38:35.183171 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:35.183274 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:35.183476 (+ 202us) server_negotiation.cc:299] Negotiation successful
0620 18:38:35.183532 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 18:38:35.184446 28969 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:38:35.273192 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 28847
I20260620 18:38:35.286594 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 28977
I20260620 18:38:35.289417 29119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:35.287542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50096 (local address 127.23.115.3:40909)
0620 18:38:35.287687 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:35.287692 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:35.287708 (+ 16us) server_negotiation.cc:407] Connection header received
0620 18:38:35.287760 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:35.287764 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:35.287835 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:35.287917 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:35.288838 (+ 921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:35.288843 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:35.288846 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:38:35.288863 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:35.288895 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:38:35.288906 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:35.289026 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:35.289088 (+ 62us) server_negotiation.cc:299] Negotiation successful
0620 18:38:35.289134 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 18:38:35.290069 29099 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:38:35.381623 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 28977
I20260620 18:38:35.396579 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 28626
I20260620 18:38:35.398110 28687 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:38:35.503974 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 28626
2026-06-20T18:38:35Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2694 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260620 18:38:35.544266 24012 mini_oidc.cc:152] Starting JWKS server
I20260620 18:38:35.546881 24012 webserver.cc:460] Webserver started at https://127.0.0.1:43123/ using document root <none> and password file <none>
I20260620 18:38:35.547233 24012 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 18:38:35.547883 24012 webserver.cc:460] Webserver started at http://127.0.0.1:37307/ using document root <none> and password file <none>
2026-06-20T18:38:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:38:35Z Disabled control of system clock
I20260620 18:38:35.572309 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980676253068-24012-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980676253068-24012-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980676253068-24012-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.115.62:34283
--webserver_interface=127.23.115.62
--webserver_port=0
--builtin_ntp_servers=127.23.115.20:42121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.115.62:34283
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43123/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980676253068-24012-0/testchainca.cert with env {}
W20260620 18:38:35.885432 29159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:35.885792 29159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:35.885913 29159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:35.894371 29159 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:38:35.894528 29159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:35.894595 29159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:35.894676 29159 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:38:35.894737 29159 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 18:38:35.894825 29159 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 18:38:35.894879 29159 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43123/jwks.json
I20260620 18:38:35.907615 29159 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:42121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980676253068-24012-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980676253068-24012-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.115.62:34283
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.115.62:34283
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43123/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.115.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980676253068-24012-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980676253068-24012-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:38:35.908994 29159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:35.910291 29159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:35.923074 29164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:35.923144 29165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:35.923249 29167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:35.924790 29159 server_base.cc:1034] running on GCE node
I20260620 18:38:35.926165 29159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:35.927917 29159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:35.929165 29159 hybrid_clock.cc:648] HybridClock initialized: now 1781980715929110 us; error 53 us; skew 500 ppm
I20260620 18:38:35.929539 29159 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:38:35.932472 29159 webserver.cc:460] Webserver started at http://127.23.115.62:43325/ using document root <none> and password file <none>
I20260620 18:38:35.933530 29159 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:35.933679 29159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:35.934065 29159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:38:35.937621 29159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980676253068-24012-0/minicluster-data/master-0/data/instance:
uuid: "c6b1d278ac3349d1b1bd040bef2ed21a"
format_stamp: "Formatted at 2026-06-20 18:38:35 on dist-test-slave-d645"
server_key: "c8b1604214e6193259195e7cce5b17e9"
server_key_iv: "1f690bcc2636f87ee1e63585e161a0fd"
server_key_version: "encryptionkey@0"
I20260620 18:38:35.938488 29159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980676253068-24012-0/minicluster-data/master-0/wal/instance:
uuid: "c6b1d278ac3349d1b1bd040bef2ed21a"
format_stamp: "Formatted at 2026-06-20 18:38:35 on dist-test-slave-d645"
server_key: "c8b1604214e6193259195e7cce5b17e9"
server_key_iv: "1f690bcc2636f87ee1e63585e161a0fd"
server_key_version: "encryptionkey@0"
I20260620 18:38:35.944228 29159 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 18:38:35.947882 29173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:35.949411 29159 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:38:35.949589 29159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980676253068-24012-0/minicluster-data/master-0/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980676253068-24012-0/minicluster-data/master-0/wal
uuid: "c6b1d278ac3349d1b1bd040bef2ed21a"
format_stamp: "Formatted at 2026-06-20 18:38:35 on dist-test-slave-d645"
server_key: "c8b1604214e6193259195e7cce5b17e9"
server_key_iv: "1f690bcc2636f87ee1e63585e161a0fd"
server_key_version: "encryptionkey@0"
I20260620 18:38:35.949808 29159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980676253068-24012-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980676253068-24012-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980676253068-24012-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:35.967610 29159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:35.968494 29159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:35.968777 29159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:35.988268 29159 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.62:34283
I20260620 18:38:35.988277 29224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.62:34283 every 8 connection(s)
I20260620 18:38:35.989581 29159 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
I20260620 18:38:35.990777 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 29159
I20260620 18:38:35.991086 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980676253068-24012-0/minicluster-data/master-0/wal/instance
I20260620 18:38:35.991864 24012 external_mini_cluster.cc:1428] Setting key e29b4a683ecc331873337456e4713dc3
I20260620 18:38:35.994580 29225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:36.006847 29225 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6b1d278ac3349d1b1bd040bef2ed21a: Bootstrap starting.
I20260620 18:38:36.007470 29228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:35.993838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38462 (local address 127.23.115.62:34283)
0620 18:38:35.994892 (+ 1054us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:35.994903 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:35.994970 (+ 67us) server_negotiation.cc:407] Connection header received
0620 18:38:35.996615 (+ 1645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:35.996637 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:35.997098 (+ 461us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:35.997475 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:35.998014 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:35.999832 (+ 1818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:36.001231 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:36.002090 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:36.002283 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:36.004220 (+ 1937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:36.004232 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:36.004261 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:38:36.005231 (+ 970us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:36.005341 (+ 110us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:38:36.005376 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:36.005546 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:36.005863 (+ 317us) server_negotiation.cc:299] Negotiation successful
0620 18:38:36.006370 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":367,"threads_started":1}
I20260620 18:38:36.011513 29225 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c6b1d278ac3349d1b1bd040bef2ed21a: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:36.011891 24012 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:38:36.012624 29225 log.cc:824] T 00000000000000000000000000000000 P c6b1d278ac3349d1b1bd040bef2ed21a: Log is configured to *not* fsync() on all Append() calls
I20260620 18:38:36.015481 29225 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6b1d278ac3349d1b1bd040bef2ed21a: No bootstrap required, opened a new log
I20260620 18:38:36.027379 29225 raft_consensus.cc:348] T 00000000000000000000000000000000 P c6b1d278ac3349d1b1bd040bef2ed21a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6b1d278ac3349d1b1bd040bef2ed21a" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 34283 } }
I20260620 18:38:36.027797 29225 raft_consensus.cc:374] T 00000000000000000000000000000000 P c6b1d278ac3349d1b1bd040bef2ed21a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:36.027892 29225 raft_consensus.cc:729] T 00000000000000000000000000000000 P c6b1d278ac3349d1b1bd040bef2ed21a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6b1d278ac3349d1b1bd040bef2ed21a, State: Initialized, Role: FOLLOWER
I20260620 18:38:36.028432 29225 consensus_queue.cc:260] T 00000000000000000000000000000000 P c6b1d278ac3349d1b1bd040bef2ed21a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6b1d278ac3349d1b1bd040bef2ed21a" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 34283 } }
I20260620 18:38:36.028667 29225 raft_consensus.cc:388] T 00000000000000000000000000000000 P c6b1d278ac3349d1b1bd040bef2ed21a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:38:36.028792 29225 raft_consensus.cc:482] T 00000000000000000000000000000000 P c6b1d278ac3349d1b1bd040bef2ed21a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:38:36.028954 29225 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c6b1d278ac3349d1b1bd040bef2ed21a [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:36.031915 29225 raft_consensus.cc:504] T 00000000000000000000000000000000 P c6b1d278ac3349d1b1bd040bef2ed21a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6b1d278ac3349d1b1bd040bef2ed21a" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 34283 } }
I20260620 18:38:36.032713 29225 leader_election.cc:302] T 00000000000000000000000000000000 P c6b1d278ac3349d1b1bd040bef2ed21a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c6b1d278ac3349d1b1bd040bef2ed21a; no voters:
I20260620 18:38:36.033447 29225 leader_election.cc:288] T 00000000000000000000000000000000 P c6b1d278ac3349d1b1bd040bef2ed21a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:38:36.033601 29230 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c6b1d278ac3349d1b1bd040bef2ed21a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:38:36.034744 29230 raft_consensus.cc:686] T 00000000000000000000000000000000 P c6b1d278ac3349d1b1bd040bef2ed21a [term 1 LEADER]: Becoming Leader. State: Replica: c6b1d278ac3349d1b1bd040bef2ed21a, State: Running, Role: LEADER
I20260620 18:38:36.035283 29230 consensus_queue.cc:237] T 00000000000000000000000000000000 P c6b1d278ac3349d1b1bd040bef2ed21a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6b1d278ac3349d1b1bd040bef2ed21a" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 34283 } }
I20260620 18:38:36.035718 29225 sys_catalog.cc:564] T 00000000000000000000000000000000 P c6b1d278ac3349d1b1bd040bef2ed21a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:38:36.037878 29232 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6b1d278ac3349d1b1bd040bef2ed21a [sys.catalog]: SysCatalogTable state changed. Reason: New leader c6b1d278ac3349d1b1bd040bef2ed21a. Latest consensus state: current_term: 1 leader_uuid: "c6b1d278ac3349d1b1bd040bef2ed21a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6b1d278ac3349d1b1bd040bef2ed21a" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 34283 } } }
I20260620 18:38:36.037972 29231 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6b1d278ac3349d1b1bd040bef2ed21a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c6b1d278ac3349d1b1bd040bef2ed21a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6b1d278ac3349d1b1bd040bef2ed21a" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 34283 } } }
I20260620 18:38:36.038318 29232 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6b1d278ac3349d1b1bd040bef2ed21a [sys.catalog]: This master's current role is: LEADER
I20260620 18:38:36.038347 29231 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6b1d278ac3349d1b1bd040bef2ed21a [sys.catalog]: This master's current role is: LEADER
I20260620 18:38:36.039198 29238 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:38:36.044901 29238 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:38:36.058081 29238 catalog_manager.cc:1347] Generated new cluster ID: 14dd503a29dd498ab7754989fefcd4a0
I20260620 18:38:36.058239 29238 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:38:36.072520 29238 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:38:36.075053 29238 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:38:36.088528 29238 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c6b1d278ac3349d1b1bd040bef2ed21a: Generated new TSK 0
I20260620 18:38:36.089581 29238 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 18:38:36.139562 29255 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260620 18:38:36.139746 29255 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 18:38:36.141131 29228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:36.117811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38466 (local address 127.23.115.62:34283)
0620 18:38:36.118029 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:36.118035 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:36.118478 (+ 443us) server_negotiation.cc:407] Connection header received
0620 18:38:36.118837 (+ 359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:36.118841 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:36.118921 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:36.119100 (+ 179us) server_negotiation.cc:226] Negotiated authn=JWT
0620 18:38:36.119676 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:36.120435 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:36.122722 (+ 2287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:36.123235 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:36.123331 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:36.124027 (+ 696us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 18:38:36.140305 (+ 16278us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43123/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 18:38:36.140813 (+ 508us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38466: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43123/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":66}
W20260620 18:38:36.141479 29228 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38466: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43123/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260620 18:38:36.143312 29254 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:38:36.117708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:34283 (local address 127.0.0.1:38466)
0620 18:38:36.118364 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:36.118383 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:36.118579 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:36.119121 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:36.119126 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:36.119139 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0620 18:38:36.119526 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:36.119538 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:36.120564 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:36.120569 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:36.122547 (+ 1978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:36.122563 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:36.123415 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:36.123422 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:36.123617 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:36.123696 (+ 79us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 18:38:36.140746 (+ 17050us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43123/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 18:38:36.141152 (+ 406us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.23.115.62:34283: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43123/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":301,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.23.115.62:34283: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43123/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260620 18:38:36.144816 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 29159
I20260620 18:38:36.146216 29220 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:38:36.239601 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 29159
2026-06-20T18:38:36Z chronyd exiting
I20260620 18:38:36.287264 24012 test_util.cc:175] -----------------------------------------------
I20260620 18:38:36.287412 24012 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980676253068-24012-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (748 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260620 18:38:36.291167 24012 mini_oidc.cc:152] Starting JWKS server
I20260620 18:38:36.292996 24012 webserver.cc:460] Webserver started at https://127.0.0.1:38611/ using document root <none> and password file <none>
I20260620 18:38:36.293201 24012 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 18:38:36.293953 24012 webserver.cc:460] Webserver started at http://127.0.0.1:38373/ using document root <none> and password file <none>
2026-06-20T18:38:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:38:36Z Disabled control of system clock
I20260620 18:38:36.326467 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980676253068-24012-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980676253068-24012-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980676253068-24012-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.115.62:32863
--webserver_interface=127.23.115.62
--webserver_port=0
--builtin_ntp_servers=127.23.115.20:43431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.115.62:32863
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38611/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980676253068-24012-0/testchainca.cert with env {}
W20260620 18:38:36.646430 29262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:36.646742 29262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:36.646864 29262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:36.654485 29262 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:38:36.654601 29262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:36.654660 29262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:36.654738 29262 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:38:36.654779 29262 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 18:38:36.654844 29262 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 18:38:36.654906 29262 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38611/jwks.json
I20260620 18:38:36.665452 29262 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:43431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980676253068-24012-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980676253068-24012-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.115.62:32863
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.115.62:32863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38611/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.115.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980676253068-24012-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980676253068-24012-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:38:36.666805 29262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:36.668143 29262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:36.681031 29267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:36.680994 29270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:36.681088 29268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:36.682353 29262 server_base.cc:1034] running on GCE node
I20260620 18:38:36.683212 29262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:36.684640 29262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:36.685860 29262 hybrid_clock.cc:648] HybridClock initialized: now 1781980716685812 us; error 34 us; skew 500 ppm
I20260620 18:38:36.686266 29262 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:38:36.691944 29262 webserver.cc:460] Webserver started at http://127.23.115.62:42073/ using document root <none> and password file <none>
I20260620 18:38:36.692646 29262 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:36.692744 29262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:36.693054 29262 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:38:36.696136 29262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980676253068-24012-0/minicluster-data/master-0/data/instance:
uuid: "b1b580e9cd0541fdb88eff757ec26f85"
format_stamp: "Formatted at 2026-06-20 18:38:36 on dist-test-slave-d645"
server_key: "99a14e35129643f3bdcaebc37aa075ab"
server_key_iv: "3890bb1041a6c2b1c41f7a6df006273a"
server_key_version: "encryptionkey@0"
I20260620 18:38:36.696949 29262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980676253068-24012-0/minicluster-data/master-0/wal/instance:
uuid: "b1b580e9cd0541fdb88eff757ec26f85"
format_stamp: "Formatted at 2026-06-20 18:38:36 on dist-test-slave-d645"
server_key: "99a14e35129643f3bdcaebc37aa075ab"
server_key_iv: "3890bb1041a6c2b1c41f7a6df006273a"
server_key_version: "encryptionkey@0"
I20260620 18:38:36.702225 29262 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260620 18:38:36.705987 29276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:36.707813 29262 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:38:36.708015 29262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980676253068-24012-0/minicluster-data/master-0/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980676253068-24012-0/minicluster-data/master-0/wal
uuid: "b1b580e9cd0541fdb88eff757ec26f85"
format_stamp: "Formatted at 2026-06-20 18:38:36 on dist-test-slave-d645"
server_key: "99a14e35129643f3bdcaebc37aa075ab"
server_key_iv: "3890bb1041a6c2b1c41f7a6df006273a"
server_key_version: "encryptionkey@0"
I20260620 18:38:36.708241 29262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980676253068-24012-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980676253068-24012-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980676253068-24012-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:36.737578 29262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:36.739306 29262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:36.739671 29262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:36.760859 29262 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.62:32863
I20260620 18:38:36.760952 29327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.62:32863 every 8 connection(s)
I20260620 18:38:36.762219 29262 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
I20260620 18:38:36.766161 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 29262
I20260620 18:38:36.766320 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980676253068-24012-0/minicluster-data/master-0/wal/instance
I20260620 18:38:36.766570 29328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:36.766997 24012 external_mini_cluster.cc:1428] Setting key b38b641f38bc69d997e0c1e9508a5f81
I20260620 18:38:36.780803 29328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1b580e9cd0541fdb88eff757ec26f85: Bootstrap starting.
I20260620 18:38:36.781172 29331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:36.768767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33736 (local address 127.23.115.62:32863)
0620 18:38:36.769796 (+ 1029us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:36.769807 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:36.769858 (+ 51us) server_negotiation.cc:407] Connection header received
0620 18:38:36.771482 (+ 1624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:36.771503 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:36.771952 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:36.772307 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:36.772871 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:36.774501 (+ 1630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:36.776146 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:36.777004 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:36.777282 (+ 278us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:36.778679 (+ 1397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:36.778691 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:36.778717 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:38:36.779370 (+ 653us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:36.779446 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:38:36.779467 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:36.779560 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:36.779875 (+ 315us) server_negotiation.cc:299] Negotiation successful
0620 18:38:36.780205 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":386,"threads_started":1}
I20260620 18:38:36.784268 29328 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b1b580e9cd0541fdb88eff757ec26f85: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:36.785382 24012 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:38:36.785429 29328 log.cc:824] T 00000000000000000000000000000000 P b1b580e9cd0541fdb88eff757ec26f85: Log is configured to *not* fsync() on all Append() calls
I20260620 18:38:36.788515 29328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1b580e9cd0541fdb88eff757ec26f85: No bootstrap required, opened a new log
I20260620 18:38:36.795888 29328 raft_consensus.cc:348] T 00000000000000000000000000000000 P b1b580e9cd0541fdb88eff757ec26f85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1b580e9cd0541fdb88eff757ec26f85" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 32863 } }
I20260620 18:38:36.796277 29328 raft_consensus.cc:374] T 00000000000000000000000000000000 P b1b580e9cd0541fdb88eff757ec26f85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:36.796363 29328 raft_consensus.cc:729] T 00000000000000000000000000000000 P b1b580e9cd0541fdb88eff757ec26f85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1b580e9cd0541fdb88eff757ec26f85, State: Initialized, Role: FOLLOWER
I20260620 18:38:36.796934 29328 consensus_queue.cc:260] T 00000000000000000000000000000000 P b1b580e9cd0541fdb88eff757ec26f85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1b580e9cd0541fdb88eff757ec26f85" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 32863 } }
I20260620 18:38:36.797163 29328 raft_consensus.cc:388] T 00000000000000000000000000000000 P b1b580e9cd0541fdb88eff757ec26f85 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:38:36.797282 29328 raft_consensus.cc:482] T 00000000000000000000000000000000 P b1b580e9cd0541fdb88eff757ec26f85 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:38:36.797428 29328 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b1b580e9cd0541fdb88eff757ec26f85 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:36.799615 29328 raft_consensus.cc:504] T 00000000000000000000000000000000 P b1b580e9cd0541fdb88eff757ec26f85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1b580e9cd0541fdb88eff757ec26f85" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 32863 } }
I20260620 18:38:36.800099 29328 leader_election.cc:302] T 00000000000000000000000000000000 P b1b580e9cd0541fdb88eff757ec26f85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b1b580e9cd0541fdb88eff757ec26f85; no voters:
I20260620 18:38:36.800642 29328 leader_election.cc:288] T 00000000000000000000000000000000 P b1b580e9cd0541fdb88eff757ec26f85 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:38:36.800783 29333 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b1b580e9cd0541fdb88eff757ec26f85 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:38:36.801457 29333 raft_consensus.cc:686] T 00000000000000000000000000000000 P b1b580e9cd0541fdb88eff757ec26f85 [term 1 LEADER]: Becoming Leader. State: Replica: b1b580e9cd0541fdb88eff757ec26f85, State: Running, Role: LEADER
I20260620 18:38:36.802022 29333 consensus_queue.cc:237] T 00000000000000000000000000000000 P b1b580e9cd0541fdb88eff757ec26f85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1b580e9cd0541fdb88eff757ec26f85" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 32863 } }
I20260620 18:38:36.802654 29328 sys_catalog.cc:564] T 00000000000000000000000000000000 P b1b580e9cd0541fdb88eff757ec26f85 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:38:36.804836 29335 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1b580e9cd0541fdb88eff757ec26f85 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b1b580e9cd0541fdb88eff757ec26f85. Latest consensus state: current_term: 1 leader_uuid: "b1b580e9cd0541fdb88eff757ec26f85" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1b580e9cd0541fdb88eff757ec26f85" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 32863 } } }
I20260620 18:38:36.805233 29335 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1b580e9cd0541fdb88eff757ec26f85 [sys.catalog]: This master's current role is: LEADER
I20260620 18:38:36.804852 29334 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1b580e9cd0541fdb88eff757ec26f85 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b1b580e9cd0541fdb88eff757ec26f85" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1b580e9cd0541fdb88eff757ec26f85" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 32863 } } }
I20260620 18:38:36.807116 29334 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1b580e9cd0541fdb88eff757ec26f85 [sys.catalog]: This master's current role is: LEADER
I20260620 18:38:36.807994 29339 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:38:36.813086 29339 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:38:36.822711 29339 catalog_manager.cc:1347] Generated new cluster ID: 8a218946523b4cb9b8d9092c470edb8b
I20260620 18:38:36.822837 29339 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:38:36.844091 29339 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:38:36.845992 29339 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:38:36.862885 29339 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b1b580e9cd0541fdb88eff757ec26f85: Generated new TSK 0
I20260620 18:38:36.864068 29339 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 18:38:36.899185 29358 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260620 18:38:36.899296 29358 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 18:38:36.900668 29331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:36.880245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33742 (local address 127.23.115.62:32863)
0620 18:38:36.880440 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:36.880447 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:36.880799 (+ 352us) server_negotiation.cc:407] Connection header received
0620 18:38:36.881001 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:36.881007 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:36.881108 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:36.881279 (+ 171us) server_negotiation.cc:226] Negotiated authn=JWT
0620 18:38:36.881695 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:36.882801 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:36.884423 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:36.885127 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:36.885253 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:36.885757 (+ 504us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 18:38:36.899944 (+ 14187us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38611/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 18:38:36.900341 (+ 397us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33742: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38611/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":68}
W20260620 18:38:36.900784 29357 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:38:36.880054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:32863 (local address 127.0.0.1:33742)
0620 18:38:36.880696 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:36.880711 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:36.880840 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:36.881300 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:36.881304 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:36.881314 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0620 18:38:36.881545 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:36.881552 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:36.883027 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:36.883032 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:36.884256 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:36.884270 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:36.885298 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:36.885303 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:36.885446 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:36.885482 (+ 36us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 18:38:36.900216 (+ 14734us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38611/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 18:38:36.900447 (+ 231us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.23.115.62:32863: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38611/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":365,"threads_started":1}
W20260620 18:38:36.900796 29331 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33742: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38611/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260620 18:38:36.902173 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 29262
I20260620 18:38:36.903443 29323 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:38:36.999053 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 29262
2026-06-20T18:38:37Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (759 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260620 18:38:37.051478 24012 mini_oidc.cc:152] Starting JWKS server
I20260620 18:38:37.053320 24012 webserver.cc:460] Webserver started at https://127.0.0.1:40491/ using document root <none> and password file <none>
I20260620 18:38:37.053550 24012 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 18:38:37.054199 24012 webserver.cc:460] Webserver started at http://127.0.0.1:35401/ using document root <none> and password file <none>
2026-06-20T18:38:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:38:37Z Disabled control of system clock
I20260620 18:38:37.078806 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980676253068-24012-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980676253068-24012-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980676253068-24012-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.115.62:36187
--webserver_interface=127.23.115.62
--webserver_port=0
--builtin_ntp_servers=127.23.115.20:35681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.115.62:36187
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40491/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 18:38:37.395802 29365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:37.396083 29365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:37.396210 29365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:37.403421 29365 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:38:37.403529 29365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:37.403571 29365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:37.403650 29365 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:38:37.403704 29365 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 18:38:37.403789 29365 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 18:38:37.403836 29365 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40491/jwks.json
I20260620 18:38:37.412835 29365 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:35681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980676253068-24012-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980676253068-24012-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.115.62:36187
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.115.62:36187
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40491/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.115.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980676253068-24012-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:38:37.414049 29365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:37.415750 29365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:37.426681 29370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:37.426726 29371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:37.426751 29373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:37.427634 29365 server_base.cc:1034] running on GCE node
I20260620 18:38:37.428375 29365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:37.429785 29365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:37.431006 29365 hybrid_clock.cc:648] HybridClock initialized: now 1781980717430956 us; error 54 us; skew 500 ppm
I20260620 18:38:37.431363 29365 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:38:37.433494 29365 webserver.cc:460] Webserver started at http://127.23.115.62:38341/ using document root <none> and password file <none>
I20260620 18:38:37.434137 29365 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:37.434233 29365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:37.434526 29365 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:38:37.437347 29365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980676253068-24012-0/minicluster-data/master-0/data/instance:
uuid: "643f4aa669e44c42ae472e6f9f7fbffe"
format_stamp: "Formatted at 2026-06-20 18:38:37 on dist-test-slave-d645"
server_key: "b676ff3150541122c6928d2c043a90c7"
server_key_iv: "0261eeb1f7071d4e287b0a315fb94767"
server_key_version: "encryptionkey@0"
I20260620 18:38:37.438087 29365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980676253068-24012-0/minicluster-data/master-0/wal/instance:
uuid: "643f4aa669e44c42ae472e6f9f7fbffe"
format_stamp: "Formatted at 2026-06-20 18:38:37 on dist-test-slave-d645"
server_key: "b676ff3150541122c6928d2c043a90c7"
server_key_iv: "0261eeb1f7071d4e287b0a315fb94767"
server_key_version: "encryptionkey@0"
I20260620 18:38:37.443297 29365 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 18:38:37.446619 29379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:37.448005 29365 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:38:37.448165 29365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980676253068-24012-0/minicluster-data/master-0/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980676253068-24012-0/minicluster-data/master-0/wal
uuid: "643f4aa669e44c42ae472e6f9f7fbffe"
format_stamp: "Formatted at 2026-06-20 18:38:37 on dist-test-slave-d645"
server_key: "b676ff3150541122c6928d2c043a90c7"
server_key_iv: "0261eeb1f7071d4e287b0a315fb94767"
server_key_version: "encryptionkey@0"
I20260620 18:38:37.448356 29365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980676253068-24012-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980676253068-24012-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980676253068-24012-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:37.465258 29365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:37.466043 29365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:37.466336 29365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:37.483641 29365 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.62:36187
I20260620 18:38:37.483660 29430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.62:36187 every 8 connection(s)
I20260620 18:38:37.484895 29365 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
I20260620 18:38:37.486331 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 29365
I20260620 18:38:37.486537 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980676253068-24012-0/minicluster-data/master-0/wal/instance
I20260620 18:38:37.487329 24012 external_mini_cluster.cc:1428] Setting key 9c5cd51b7a7e3b08ecb8a7062e10baed
I20260620 18:38:37.489183 29431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:37.501108 29434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:37.489225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49248 (local address 127.23.115.62:36187)
0620 18:38:37.490317 (+ 1092us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:37.490328 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:37.490413 (+ 85us) server_negotiation.cc:407] Connection header received
0620 18:38:37.492115 (+ 1702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:37.492136 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:37.492567 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:37.492874 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:37.493434 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:37.494742 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:37.496549 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:37.497368 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:37.497515 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:37.498719 (+ 1204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:37.498727 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:37.498741 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:38:37.499387 (+ 646us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:37.499475 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:38:37.499497 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:37.499608 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:37.499816 (+ 208us) server_negotiation.cc:299] Negotiation successful
0620 18:38:37.500170 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":364,"threads_started":1}
I20260620 18:38:37.502372 29431 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 643f4aa669e44c42ae472e6f9f7fbffe: Bootstrap starting.
I20260620 18:38:37.504813 24012 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:38:37.505486 29431 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 643f4aa669e44c42ae472e6f9f7fbffe: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:37.506398 29431 log.cc:824] T 00000000000000000000000000000000 P 643f4aa669e44c42ae472e6f9f7fbffe: Log is configured to *not* fsync() on all Append() calls
I20260620 18:38:37.509238 29431 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 643f4aa669e44c42ae472e6f9f7fbffe: No bootstrap required, opened a new log
I20260620 18:38:37.516810 29431 raft_consensus.cc:348] T 00000000000000000000000000000000 P 643f4aa669e44c42ae472e6f9f7fbffe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "643f4aa669e44c42ae472e6f9f7fbffe" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 36187 } }
I20260620 18:38:37.517189 29431 raft_consensus.cc:374] T 00000000000000000000000000000000 P 643f4aa669e44c42ae472e6f9f7fbffe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:37.517277 29431 raft_consensus.cc:729] T 00000000000000000000000000000000 P 643f4aa669e44c42ae472e6f9f7fbffe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 643f4aa669e44c42ae472e6f9f7fbffe, State: Initialized, Role: FOLLOWER
I20260620 18:38:37.517835 29431 consensus_queue.cc:260] T 00000000000000000000000000000000 P 643f4aa669e44c42ae472e6f9f7fbffe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "643f4aa669e44c42ae472e6f9f7fbffe" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 36187 } }
I20260620 18:38:37.518054 29431 raft_consensus.cc:388] T 00000000000000000000000000000000 P 643f4aa669e44c42ae472e6f9f7fbffe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:38:37.518184 29431 raft_consensus.cc:482] T 00000000000000000000000000000000 P 643f4aa669e44c42ae472e6f9f7fbffe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:38:37.518352 29431 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 643f4aa669e44c42ae472e6f9f7fbffe [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:37.520476 29431 raft_consensus.cc:504] T 00000000000000000000000000000000 P 643f4aa669e44c42ae472e6f9f7fbffe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "643f4aa669e44c42ae472e6f9f7fbffe" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 36187 } }
I20260620 18:38:37.520959 29431 leader_election.cc:302] T 00000000000000000000000000000000 P 643f4aa669e44c42ae472e6f9f7fbffe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 643f4aa669e44c42ae472e6f9f7fbffe; no voters:
I20260620 18:38:37.521559 29431 leader_election.cc:288] T 00000000000000000000000000000000 P 643f4aa669e44c42ae472e6f9f7fbffe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:38:37.521710 29436 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 643f4aa669e44c42ae472e6f9f7fbffe [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:38:37.522363 29436 raft_consensus.cc:686] T 00000000000000000000000000000000 P 643f4aa669e44c42ae472e6f9f7fbffe [term 1 LEADER]: Becoming Leader. State: Replica: 643f4aa669e44c42ae472e6f9f7fbffe, State: Running, Role: LEADER
I20260620 18:38:37.522881 29436 consensus_queue.cc:237] T 00000000000000000000000000000000 P 643f4aa669e44c42ae472e6f9f7fbffe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "643f4aa669e44c42ae472e6f9f7fbffe" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 36187 } }
I20260620 18:38:37.523599 29431 sys_catalog.cc:564] T 00000000000000000000000000000000 P 643f4aa669e44c42ae472e6f9f7fbffe [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:38:37.525453 29438 sys_catalog.cc:455] T 00000000000000000000000000000000 P 643f4aa669e44c42ae472e6f9f7fbffe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 643f4aa669e44c42ae472e6f9f7fbffe. Latest consensus state: current_term: 1 leader_uuid: "643f4aa669e44c42ae472e6f9f7fbffe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "643f4aa669e44c42ae472e6f9f7fbffe" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 36187 } } }
I20260620 18:38:37.525857 29438 sys_catalog.cc:458] T 00000000000000000000000000000000 P 643f4aa669e44c42ae472e6f9f7fbffe [sys.catalog]: This master's current role is: LEADER
I20260620 18:38:37.525696 29437 sys_catalog.cc:455] T 00000000000000000000000000000000 P 643f4aa669e44c42ae472e6f9f7fbffe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "643f4aa669e44c42ae472e6f9f7fbffe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "643f4aa669e44c42ae472e6f9f7fbffe" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 36187 } } }
I20260620 18:38:37.525954 29437 sys_catalog.cc:458] T 00000000000000000000000000000000 P 643f4aa669e44c42ae472e6f9f7fbffe [sys.catalog]: This master's current role is: LEADER
I20260620 18:38:37.527719 29444 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:38:37.532828 29444 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:38:37.540934 29444 catalog_manager.cc:1347] Generated new cluster ID: 9d090b21f8724095adfb18c84c30ff01
I20260620 18:38:37.541043 29444 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:38:37.558501 29444 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:38:37.559847 29444 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:38:37.566143 29444 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 643f4aa669e44c42ae472e6f9f7fbffe: Generated new TSK 0
I20260620 18:38:37.567001 29444 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:38:37.588892 29434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:37.580486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57078 (local address 127.23.115.62:36187)
0620 18:38:37.580680 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:37.580687 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:37.581162 (+ 475us) server_negotiation.cc:407] Connection header received
0620 18:38:37.581376 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:37.581381 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:37.581474 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:37.581643 (+ 169us) server_negotiation.cc:226] Negotiated authn=JWT
0620 18:38:37.582060 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:37.583198 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:37.584606 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:37.585267 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:37.585401 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:37.585879 (+ 478us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 18:38:37.588248 (+ 2369us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40491/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 18:38:37.588573 (+ 325us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57078: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40491/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":60}
W20260620 18:38:37.589041 29434 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57078: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40491/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260620 18:38:37.590653 29460 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:38:37.580453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:36187 (local address 127.0.0.1:57078)
0620 18:38:37.581066 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:37.581082 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:37.581200 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:37.581659 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:37.581663 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:37.581672 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0620 18:38:37.581924 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:37.581932 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:37.583352 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:37.583356 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:37.584464 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:37.584476 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:37.585449 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:37.585454 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:37.585589 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:37.585622 (+ 33us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 18:38:37.588472 (+ 2850us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40491/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 18:38:37.590273 (+ 1801us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.23.115.62:36187: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40491/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":478,"mutex_wait_us":1488,"thread_start_us":318,"threads_started":1}
I20260620 18:38:37.591692 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 29365
I20260620 18:38:37.593048 29426 generic_service.cc:196] Checking for leaks (request via RPC)
E20260620 18:38:37.600312 29461 webserver.cc:533] Webserver: sslize failed:
E20260620 18:38:37.600399 29461 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 18:38:37.683614 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 29365
2026-06-20T18:38:37Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (684 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-20T18:38:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:38:37Z Disabled control of system clock
I20260620 18:38:37.760315 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980676253068-24012-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980676253068-24012-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980676253068-24012-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.115.62:42527
--webserver_interface=127.23.115.62
--webserver_port=0
--builtin_ntp_servers=127.23.115.20:35189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.115.62:42527
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980676253068-24012-0/testchainca.cert with env {}
W20260620 18:38:38.077843 29466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:38.078158 29466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:38.078276 29466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:38.088265 29466 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:38:38.088377 29466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:38.088415 29466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:38.088491 29466 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:38:38.088548 29466 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 18:38:38.088630 29466 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 18:38:38.088680 29466 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260620 18:38:38.098650 29466 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:35189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980676253068-24012-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980676253068-24012-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.115.62:42527
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.115.62:42527
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.115.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980676253068-24012-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980676253068-24012-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:38:38.099941 29466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:38.101199 29466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:38.112686 29471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:38.112644 29472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:38.112612 29474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:38.113548 29466 server_base.cc:1034] running on GCE node
I20260620 18:38:38.114295 29466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:38.115724 29466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:38.116967 29466 hybrid_clock.cc:648] HybridClock initialized: now 1781980718116897 us; error 57 us; skew 500 ppm
I20260620 18:38:38.117353 29466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:38:38.119782 29466 webserver.cc:460] Webserver started at http://127.23.115.62:43757/ using document root <none> and password file <none>
I20260620 18:38:38.120502 29466 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:38.120610 29466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:38.120900 29466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:38:38.123790 29466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980676253068-24012-0/minicluster-data/master-0/data/instance:
uuid: "ad2c375ffeb3460fac14a50127a7b3dd"
format_stamp: "Formatted at 2026-06-20 18:38:38 on dist-test-slave-d645"
server_key: "fe02b1935bca2972fb08eedf0608172f"
server_key_iv: "dd02304d4c251a1b6857f44f13dfa60d"
server_key_version: "encryptionkey@0"
I20260620 18:38:38.124577 29466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980676253068-24012-0/minicluster-data/master-0/wal/instance:
uuid: "ad2c375ffeb3460fac14a50127a7b3dd"
format_stamp: "Formatted at 2026-06-20 18:38:38 on dist-test-slave-d645"
server_key: "fe02b1935bca2972fb08eedf0608172f"
server_key_iv: "dd02304d4c251a1b6857f44f13dfa60d"
server_key_version: "encryptionkey@0"
I20260620 18:38:38.129885 29466 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 18:38:38.133764 29480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:38.135653 29466 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 18:38:38.135823 29466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980676253068-24012-0/minicluster-data/master-0/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980676253068-24012-0/minicluster-data/master-0/wal
uuid: "ad2c375ffeb3460fac14a50127a7b3dd"
format_stamp: "Formatted at 2026-06-20 18:38:38 on dist-test-slave-d645"
server_key: "fe02b1935bca2972fb08eedf0608172f"
server_key_iv: "dd02304d4c251a1b6857f44f13dfa60d"
server_key_version: "encryptionkey@0"
I20260620 18:38:38.136040 29466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980676253068-24012-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980676253068-24012-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980676253068-24012-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:38.151211 29466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:38.152000 29466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:38.152230 29466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:38.170207 29466 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.62:42527
I20260620 18:38:38.170264 29531 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.62:42527 every 8 connection(s)
I20260620 18:38:38.171540 29466 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
I20260620 18:38:38.175473 29532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:38.180356 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 29466
I20260620 18:38:38.180578 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980676253068-24012-0/minicluster-data/master-0/wal/instance
I20260620 18:38:38.181420 24012 external_mini_cluster.cc:1428] Setting key d4289bb971e00358d122c4f52c223d05
I20260620 18:38:38.188621 29532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad2c375ffeb3460fac14a50127a7b3dd: Bootstrap starting.
I20260620 18:38:38.192646 29532 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ad2c375ffeb3460fac14a50127a7b3dd: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:38.193758 29532 log.cc:824] T 00000000000000000000000000000000 P ad2c375ffeb3460fac14a50127a7b3dd: Log is configured to *not* fsync() on all Append() calls
I20260620 18:38:38.197667 29532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad2c375ffeb3460fac14a50127a7b3dd: No bootstrap required, opened a new log
I20260620 18:38:38.199048 29535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:38.184373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44686 (local address 127.23.115.62:42527)
0620 18:38:38.185622 (+ 1249us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:38.185639 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:38.185697 (+ 58us) server_negotiation.cc:407] Connection header received
0620 18:38:38.188258 (+ 2561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:38.188288 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:38.188911 (+ 623us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:38.189371 (+ 460us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:38.189946 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:38.192026 (+ 2080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:38.193461 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:38.194616 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:38.194762 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:38.196124 (+ 1362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:38.196135 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:38.196167 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:38:38.196932 (+ 765us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:38.197041 (+ 109us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:38:38.197067 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:38.197189 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:38.197405 (+ 216us) server_negotiation.cc:299] Negotiation successful
0620 18:38:38.197747 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":655,"thread_start_us":391,"threads_started":1}
I20260620 18:38:38.203487 24012 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:38:38.207890 29532 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad2c375ffeb3460fac14a50127a7b3dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad2c375ffeb3460fac14a50127a7b3dd" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 42527 } }
I20260620 18:38:38.208272 29532 raft_consensus.cc:374] T 00000000000000000000000000000000 P ad2c375ffeb3460fac14a50127a7b3dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:38.208360 29532 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad2c375ffeb3460fac14a50127a7b3dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad2c375ffeb3460fac14a50127a7b3dd, State: Initialized, Role: FOLLOWER
I20260620 18:38:38.208865 29532 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad2c375ffeb3460fac14a50127a7b3dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad2c375ffeb3460fac14a50127a7b3dd" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 42527 } }
I20260620 18:38:38.209057 29532 raft_consensus.cc:388] T 00000000000000000000000000000000 P ad2c375ffeb3460fac14a50127a7b3dd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:38:38.209162 29532 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad2c375ffeb3460fac14a50127a7b3dd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:38:38.209290 29532 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad2c375ffeb3460fac14a50127a7b3dd [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:38.211462 29532 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad2c375ffeb3460fac14a50127a7b3dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad2c375ffeb3460fac14a50127a7b3dd" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 42527 } }
I20260620 18:38:38.211977 29532 leader_election.cc:302] T 00000000000000000000000000000000 P ad2c375ffeb3460fac14a50127a7b3dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ad2c375ffeb3460fac14a50127a7b3dd; no voters:
I20260620 18:38:38.212711 29532 leader_election.cc:288] T 00000000000000000000000000000000 P ad2c375ffeb3460fac14a50127a7b3dd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:38:38.212837 29537 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad2c375ffeb3460fac14a50127a7b3dd [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:38:38.213469 29537 raft_consensus.cc:686] T 00000000000000000000000000000000 P ad2c375ffeb3460fac14a50127a7b3dd [term 1 LEADER]: Becoming Leader. State: Replica: ad2c375ffeb3460fac14a50127a7b3dd, State: Running, Role: LEADER
I20260620 18:38:38.214072 29537 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad2c375ffeb3460fac14a50127a7b3dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad2c375ffeb3460fac14a50127a7b3dd" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 42527 } }
I20260620 18:38:38.214998 29532 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad2c375ffeb3460fac14a50127a7b3dd [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:38:38.216980 29539 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad2c375ffeb3460fac14a50127a7b3dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad2c375ffeb3460fac14a50127a7b3dd. Latest consensus state: current_term: 1 leader_uuid: "ad2c375ffeb3460fac14a50127a7b3dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad2c375ffeb3460fac14a50127a7b3dd" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 42527 } } }
I20260620 18:38:38.217680 29539 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad2c375ffeb3460fac14a50127a7b3dd [sys.catalog]: This master's current role is: LEADER
I20260620 18:38:38.217160 29538 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad2c375ffeb3460fac14a50127a7b3dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ad2c375ffeb3460fac14a50127a7b3dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad2c375ffeb3460fac14a50127a7b3dd" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 42527 } } }
I20260620 18:38:38.218529 29538 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad2c375ffeb3460fac14a50127a7b3dd [sys.catalog]: This master's current role is: LEADER
I20260620 18:38:38.220008 29545 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:38:38.225828 29545 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:38:38.235401 29545 catalog_manager.cc:1347] Generated new cluster ID: bd81e126656149eeb91bc8f2e4df9caa
I20260620 18:38:38.235517 29545 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:38:38.264528 29545 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:38:38.265965 29545 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:38:38.278070 29545 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ad2c375ffeb3460fac14a50127a7b3dd: Generated new TSK 0
I20260620 18:38:38.279402 29545 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:38:38.388624 29535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:38.304583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44700 (local address 127.23.115.62:42527)
0620 18:38:38.304783 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:38.304790 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:38.305153 (+ 363us) server_negotiation.cc:407] Connection header received
0620 18:38:38.305375 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:38.305381 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:38.305481 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:38.305658 (+ 177us) server_negotiation.cc:226] Negotiated authn=JWT
0620 18:38:38.306066 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:38.307271 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:38.308708 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:38.309410 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:38.309541 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:38.310016 (+ 475us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 18:38:38.387877 (+ 77861us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 18:38:38.388286 (+ 409us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44700: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":68}
W20260620 18:38:38.388774 29535 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44700: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260620 18:38:38.388887 29561 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:38:38.304465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:42527 (local address 127.0.0.1:44700)
0620 18:38:38.305052 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:38.305068 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:38.305198 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:38.305685 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:38.305689 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:38.305699 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0620 18:38:38.305933 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:38.305940 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:38.307404 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:38.307408 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:38.308569 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:38.308584 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:38.309586 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:38.309592 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:38.309719 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:38.309761 (+ 42us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 18:38:38.388196 (+ 78435us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 18:38:38.388490 (+ 294us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.23.115.62:42527: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":327,"threads_started":1}
I20260620 18:38:38.389964 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 29466
I20260620 18:38:38.391156 29527 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:38:38.494544 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 29466
2026-06-20T18:38:38Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (791 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (402 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (417 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980676253068-24012-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:38:39 dist-test-slave-d645 krb5kdc[29576](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:38:39 dist-test-slave-d645 krb5kdc[29576](info): set up 2 sockets
Jun 20 18:38:39 dist-test-slave-d645 krb5kdc[29576](info): commencing operation
krb5kdc: starting...
W20260620 18:38:40.331477 24012 mini_kdc.cc:121] Time spent starting KDC: real 0.983s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980676253068-24012-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980676253068-24012-0/krb5kdc/test-user.keytab.
Jun 20 18:38:40 dist-test-slave-d645 krb5kdc[29576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980720, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:38:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:38:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.115.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.115.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.115.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab.
Entry for principal kudu/127.23.115.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab.
I20260620 18:38:40.532017 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980676253068-24012-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980676253068-24012-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980676253068-24012-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.115.62:33251
--webserver_interface=127.23.115.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab
--principal=kudu/127.23.115.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:44121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.115.62:33251
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:38:40.850684 29589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:40.850991 29589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:40.851130 29589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:40.858270 29589 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:38:40.858373 29589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:40.858444 29589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:40.858515 29589 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:38:40.858558 29589 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:38:40.868430 29589 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:44121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980676253068-24012-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980676253068-24012-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.115.62:33251
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab
--principal=kudu/127.23.115.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.115.62:33251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.115.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980676253068-24012-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:38:40.869700 29589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:40.870921 29589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:40.882268 29594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:40.882205 29597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:40.882244 29595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:40.882876 29589 server_base.cc:1034] running on GCE node
I20260620 18:38:40.883661 29589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:40.885419 29589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:40.886644 29589 hybrid_clock.cc:648] HybridClock initialized: now 1781980720886580 us; error 54 us; skew 500 ppm
I20260620 18:38:40.887059 29589 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:38:40 dist-test-slave-d645 krb5kdc[29576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980720, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:40.890126 29589 init.cc:377] Logged in from keytab as kudu/127.23.115.62@KRBTEST.COM (short username kudu)
I20260620 18:38:40.891696 29589 webserver.cc:460] Webserver started at http://127.23.115.62:45033/ using document root <none> and password file <none>
I20260620 18:38:40.892372 29589 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:40.892484 29589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:40.892771 29589 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:38:40.895658 29589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980676253068-24012-0/minicluster-data/master-0/data/instance:
uuid: "75d0c63baaae41c1809265cba6ddf218"
format_stamp: "Formatted at 2026-06-20 18:38:40 on dist-test-slave-d645"
server_key: "7408985bcea00455de4b1e0ff3fc82aa"
server_key_iv: "d8f7bb2cd4c3088835e9c4c7e5378e25"
server_key_version: "encryptionkey@0"
I20260620 18:38:40.896391 29589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980676253068-24012-0/minicluster-data/master-0/wal/instance:
uuid: "75d0c63baaae41c1809265cba6ddf218"
format_stamp: "Formatted at 2026-06-20 18:38:40 on dist-test-slave-d645"
server_key: "7408985bcea00455de4b1e0ff3fc82aa"
server_key_iv: "d8f7bb2cd4c3088835e9c4c7e5378e25"
server_key_version: "encryptionkey@0"
I20260620 18:38:40.901336 29589 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:38:40.904768 29604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:40.906093 29589 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 18:38:40.906293 29589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980676253068-24012-0/minicluster-data/master-0/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980676253068-24012-0/minicluster-data/master-0/wal
uuid: "75d0c63baaae41c1809265cba6ddf218"
format_stamp: "Formatted at 2026-06-20 18:38:40 on dist-test-slave-d645"
server_key: "7408985bcea00455de4b1e0ff3fc82aa"
server_key_iv: "d8f7bb2cd4c3088835e9c4c7e5378e25"
server_key_version: "encryptionkey@0"
I20260620 18:38:40.906476 29589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980676253068-24012-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980676253068-24012-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980676253068-24012-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:40.919595 29589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:40.924347 29589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:40.924616 29589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:40.943033 29589 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.62:33251
I20260620 18:38:40.943077 29655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.62:33251 every 8 connection(s)
I20260620 18:38:40.944402 29589 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
I20260620 18:38:40.947981 29656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:40.952152 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 29589
I20260620 18:38:40.952353 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980676253068-24012-0/minicluster-data/master-0/wal/instance
I20260620 18:38:40.953168 24012 external_mini_cluster.cc:1428] Setting key 5e22b271e48a2e7ff4613425d9d6a880
I20260620 18:38:40.959823 29656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75d0c63baaae41c1809265cba6ddf218: Bootstrap starting.
I20260620 18:38:40.963429 29656 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 75d0c63baaae41c1809265cba6ddf218: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:40.964516 29656 log.cc:824] T 00000000000000000000000000000000 P 75d0c63baaae41c1809265cba6ddf218: Log is configured to *not* fsync() on all Append() calls
I20260620 18:38:40.967790 29656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75d0c63baaae41c1809265cba6ddf218: No bootstrap required, opened a new log
Jun 20 18:38:40 dist-test-slave-d645 krb5kdc[29576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980720, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.115.62@KRBTEST.COM
I20260620 18:38:40.975615 29656 raft_consensus.cc:348] T 00000000000000000000000000000000 P 75d0c63baaae41c1809265cba6ddf218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75d0c63baaae41c1809265cba6ddf218" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 33251 } }
I20260620 18:38:40.975979 29656 raft_consensus.cc:374] T 00000000000000000000000000000000 P 75d0c63baaae41c1809265cba6ddf218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:40.976089 29656 raft_consensus.cc:729] T 00000000000000000000000000000000 P 75d0c63baaae41c1809265cba6ddf218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75d0c63baaae41c1809265cba6ddf218, State: Initialized, Role: FOLLOWER
I20260620 18:38:40.976496 29659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:40.955268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58998 (local address 127.23.115.62:33251)
0620 18:38:40.956056 (+ 788us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:40.956071 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:40.956120 (+ 49us) server_negotiation.cc:407] Connection header received
0620 18:38:40.958300 (+ 2180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:40.958324 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:40.958705 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:40.959056 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:40.960750 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:40.962121 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:40.963827 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:40.964514 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:40.964708 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:40.969085 (+ 4377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:40.969093 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:40.969107 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:40.969153 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:40.972161 (+ 3008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:40.972816 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:40.972825 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:40.972831 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:40.972993 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:40.973344 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:40.973347 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:40.973349 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:40.973877 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:40.974186 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:40.974582 (+ 396us) server_negotiation.cc:299] Negotiation successful
0620 18:38:40.975018 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":337,"threads_started":1}
I20260620 18:38:40.976649 29656 consensus_queue.cc:260] T 00000000000000000000000000000000 P 75d0c63baaae41c1809265cba6ddf218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75d0c63baaae41c1809265cba6ddf218" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 33251 } }
I20260620 18:38:40.976909 29656 raft_consensus.cc:388] T 00000000000000000000000000000000 P 75d0c63baaae41c1809265cba6ddf218 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:38:40.977053 29656 raft_consensus.cc:482] T 00000000000000000000000000000000 P 75d0c63baaae41c1809265cba6ddf218 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:38:40.977210 29656 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 75d0c63baaae41c1809265cba6ddf218 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:40.979627 29656 raft_consensus.cc:504] T 00000000000000000000000000000000 P 75d0c63baaae41c1809265cba6ddf218 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75d0c63baaae41c1809265cba6ddf218" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 33251 } }
I20260620 18:38:40.980098 29656 leader_election.cc:302] T 00000000000000000000000000000000 P 75d0c63baaae41c1809265cba6ddf218 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 75d0c63baaae41c1809265cba6ddf218; no voters:
I20260620 18:38:40.980643 29656 leader_election.cc:288] T 00000000000000000000000000000000 P 75d0c63baaae41c1809265cba6ddf218 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:38:40.980790 29661 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 75d0c63baaae41c1809265cba6ddf218 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:38:40.981405 29661 raft_consensus.cc:686] T 00000000000000000000000000000000 P 75d0c63baaae41c1809265cba6ddf218 [term 1 LEADER]: Becoming Leader. State: Replica: 75d0c63baaae41c1809265cba6ddf218, State: Running, Role: LEADER
I20260620 18:38:40.981957 29661 consensus_queue.cc:237] T 00000000000000000000000000000000 P 75d0c63baaae41c1809265cba6ddf218 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75d0c63baaae41c1809265cba6ddf218" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 33251 } }
I20260620 18:38:40.982556 29656 sys_catalog.cc:564] T 00000000000000000000000000000000 P 75d0c63baaae41c1809265cba6ddf218 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:38:40.985386 29663 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75d0c63baaae41c1809265cba6ddf218 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75d0c63baaae41c1809265cba6ddf218. Latest consensus state: current_term: 1 leader_uuid: "75d0c63baaae41c1809265cba6ddf218" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75d0c63baaae41c1809265cba6ddf218" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 33251 } } }
I20260620 18:38:40.986050 29663 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75d0c63baaae41c1809265cba6ddf218 [sys.catalog]: This master's current role is: LEADER
I20260620 18:38:40.985647 29662 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75d0c63baaae41c1809265cba6ddf218 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "75d0c63baaae41c1809265cba6ddf218" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75d0c63baaae41c1809265cba6ddf218" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 33251 } } }
I20260620 18:38:40.986500 29662 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75d0c63baaae41c1809265cba6ddf218 [sys.catalog]: This master's current role is: LEADER
W20260620 18:38:40.992468 29677 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 75d0c63baaae41c1809265cba6ddf218: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 18:38:40.992676 29677 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 18:38:40.993676 29674 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:38:40.995388 29674 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:38:41.004767 29674 catalog_manager.cc:1347] Generated new cluster ID: 8440237b709a4ad8ad20df613787d856
I20260620 18:38:41.004910 29674 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:38:41.021462 29674 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:38:41.023034 29674 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:38:41.033540 29674 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 75d0c63baaae41c1809265cba6ddf218: Generated new TSK 0
I20260620 18:38:41.034610 29674 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.115.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.115.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.115.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.1.keytab.
Entry for principal kudu/127.23.115.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.1.keytab.
I20260620 18:38:41.095928 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980676253068-24012-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980676253068-24012-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980676253068-24012-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980676253068-24012-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.115.1:0
--local_ip_for_outbound_sockets=127.23.115.1
--webserver_interface=127.23.115.1
--webserver_port=0
--tserver_master_addrs=127.23.115.62:33251
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.1.keytab
--principal=kudu/127.23.115.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:44121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:38:41.421216 29682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:41.421532 29682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:41.421660 29682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:41.430402 29682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:41.430558 29682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:41.430712 29682 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.115.1
I20260620 18:38:41.442046 29682 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:44121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980676253068-24012-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980676253068-24012-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.1.keytab
--principal=kudu/127.23.115.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.115.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980676253068-24012-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.115.1
--webserver_port=0
--tserver_master_addrs=127.23.115.62:33251
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.115.1
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980676253068-24012-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:38:41.443486 29682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:41.444928 29682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:41.460953 29687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:41.461020 29690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:41.461002 29688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:41.462209 29682 server_base.cc:1034] running on GCE node
I20260620 18:38:41.463294 29682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:41.464363 29682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:41.465598 29682 hybrid_clock.cc:648] HybridClock initialized: now 1781980721465548 us; error 41 us; skew 500 ppm
I20260620 18:38:41.465962 29682 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:38:41 dist-test-slave-d645 krb5kdc[29576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980721, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:41.469688 29682 init.cc:377] Logged in from keytab as kudu/127.23.115.1@KRBTEST.COM (short username kudu)
I20260620 18:38:41.471285 29682 webserver.cc:460] Webserver started at http://127.23.115.1:46877/ using document root <none> and password file <none>
I20260620 18:38:41.472137 29682 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:41.472247 29682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:41.472545 29682 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:38:41.475580 29682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980676253068-24012-0/minicluster-data/ts-0/data/instance:
uuid: "5030b7bcde024a63b6789639c658388b"
format_stamp: "Formatted at 2026-06-20 18:38:41 on dist-test-slave-d645"
server_key: "73474c7b36226fac5e0df4ec394addc7"
server_key_iv: "5f25c4e3a0666519feb56a76c2e11a0c"
server_key_version: "encryptionkey@0"
I20260620 18:38:41.476411 29682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980676253068-24012-0/minicluster-data/ts-0/wal/instance:
uuid: "5030b7bcde024a63b6789639c658388b"
format_stamp: "Formatted at 2026-06-20 18:38:41 on dist-test-slave-d645"
server_key: "73474c7b36226fac5e0df4ec394addc7"
server_key_iv: "5f25c4e3a0666519feb56a76c2e11a0c"
server_key_version: "encryptionkey@0"
I20260620 18:38:41.481721 29682 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 18:38:41.485905 29697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:41.487821 29682 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260620 18:38:41.487977 29682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980676253068-24012-0/minicluster-data/ts-0/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980676253068-24012-0/minicluster-data/ts-0/wal
uuid: "5030b7bcde024a63b6789639c658388b"
format_stamp: "Formatted at 2026-06-20 18:38:41 on dist-test-slave-d645"
server_key: "73474c7b36226fac5e0df4ec394addc7"
server_key_iv: "5f25c4e3a0666519feb56a76c2e11a0c"
server_key_version: "encryptionkey@0"
I20260620 18:38:41.488175 29682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980676253068-24012-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980676253068-24012-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980676253068-24012-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:41.501411 29682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:41.505642 29682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:41.505921 29682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:41.507040 29682 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:38:41.508476 29682 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:38:41.508548 29682 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:41.508636 29682 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:38:41.508697 29682 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:41.541771 29682 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.1:37791
I20260620 18:38:41.541843 29809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.1:37791 every 8 connection(s)
I20260620 18:38:41.543273 29682 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980676253068-24012-0/minicluster-data/ts-0/data/info.pb
I20260620 18:38:41.547642 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 29682
I20260620 18:38:41.547835 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980676253068-24012-0/minicluster-data/ts-0/wal/instance
I20260620 18:38:41.548730 24012 external_mini_cluster.cc:1428] Setting key 596d66511c0845867427dec61360f7ed
Jun 20 18:38:41 dist-test-slave-d645 krb5kdc[29576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980721, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.1@KRBTEST.COM for kudu/127.23.115.62@KRBTEST.COM
I20260620 18:38:41.565029 29812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:41.545914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.1:43631 (local address 127.23.115.62:33251)
0620 18:38:41.546350 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:41.546355 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:41.547331 (+ 976us) server_negotiation.cc:407] Connection header received
0620 18:38:41.549824 (+ 2493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:41.549829 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:41.549940 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:41.550057 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:41.551973 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:41.552690 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:41.554729 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:41.555199 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:41.555297 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:41.559067 (+ 3770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:41.559074 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:41.559079 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:41.559119 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:41.561987 (+ 2868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:41.562726 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:41.562731 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:41.562734 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:41.562864 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:41.563268 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:41.563273 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:41.563276 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:41.563601 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:41.563745 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:41.564464 (+ 719us) server_negotiation.cc:299] Negotiation successful
0620 18:38:41.564673 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":225,"threads_started":1}
I20260620 18:38:41.565860 29813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:41.546422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:33251 (local address 127.23.115.1:43631)
0620 18:38:41.547149 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:41.547213 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:41.549517 (+ 2304us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:41.550291 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:41.550300 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:41.551020 (+ 720us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:41.551801 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:41.551824 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:41.552839 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:41.552844 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:41.554551 (+ 1707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:41.554565 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:41.555408 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:41.555414 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:41.555694 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:41.556385 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:41.556413 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:41.558821 (+ 2408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:41.562149 (+ 3328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:41.562157 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:41.562171 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:41.562590 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:41.562989 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:41.562993 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:41.562995 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:41.563177 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:41.563744 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:41.563751 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:41.564070 (+ 319us) client_negotiation.cc:769] Sending connection context
0620 18:38:41.564327 (+ 257us) client_negotiation.cc:240] Negotiation successful
0620 18:38:41.564730 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":289,"threads_started":1}
I20260620 18:38:41.567392 29810 heartbeater.cc:347] Connected to a master server at 127.23.115.62:33251
I20260620 18:38:41.567677 29810 heartbeater.cc:464] Registering TS with master...
I20260620 18:38:41.568339 29810 heartbeater.cc:511] Master 127.23.115.62:33251 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.23.115.2@KRBTEST.COM; defaulting to no policy
I20260620 18:38:41.571381 29621 ts_manager.cc:194] Registered new tserver with Master: 5030b7bcde024a63b6789639c658388b (127.23.115.1:37791)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.115.2@KRBTEST.COM" created.
I20260620 18:38:41.573182 29621 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.115.1@KRBTEST.COM'} at 127.23.115.1:43631
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.115.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.2.keytab.
Entry for principal kudu/127.23.115.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.2.keytab.
I20260620 18:38:41.594152 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980676253068-24012-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980676253068-24012-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980676253068-24012-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980676253068-24012-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.115.2:0
--local_ip_for_outbound_sockets=127.23.115.2
--webserver_interface=127.23.115.2
--webserver_port=0
--tserver_master_addrs=127.23.115.62:33251
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.2.keytab
--principal=kudu/127.23.115.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:44121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:38:41.908463 29816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:41.908744 29816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:41.908838 29816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:41.916049 29816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:41.916162 29816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:41.916273 29816 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.115.2
I20260620 18:38:41.926209 29816 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:44121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980676253068-24012-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980676253068-24012-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.2.keytab
--principal=kudu/127.23.115.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.115.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980676253068-24012-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.115.2
--webserver_port=0
--tserver_master_addrs=127.23.115.62:33251
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.115.2
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980676253068-24012-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:38:41.927505 29816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:41.928707 29816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:41.941745 29824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:41.941769 29821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:41.941771 29822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:41.942713 29816 server_base.cc:1034] running on GCE node
I20260620 18:38:41.943454 29816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:41.944619 29816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:41.945894 29816 hybrid_clock.cc:648] HybridClock initialized: now 1781980721945808 us; error 76 us; skew 500 ppm
I20260620 18:38:41.946238 29816 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:38:41 dist-test-slave-d645 krb5kdc[29576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980721, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:41.952867 29816 init.cc:377] Logged in from keytab as kudu/127.23.115.2@KRBTEST.COM (short username kudu)
I20260620 18:38:41.954639 29816 webserver.cc:460] Webserver started at http://127.23.115.2:42513/ using document root <none> and password file <none>
I20260620 18:38:41.955394 29816 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:41.955500 29816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:41.955789 29816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:38:41.958576 29816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980676253068-24012-0/minicluster-data/ts-1/data/instance:
uuid: "026b703aea1d4408a40476b9ae31b4cb"
format_stamp: "Formatted at 2026-06-20 18:38:41 on dist-test-slave-d645"
server_key: "fba166657b7e62ec75111bb714148e1d"
server_key_iv: "8776d8c799cf8da9ef26c03a6674d24e"
server_key_version: "encryptionkey@0"
I20260620 18:38:41.959332 29816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980676253068-24012-0/minicluster-data/ts-1/wal/instance:
uuid: "026b703aea1d4408a40476b9ae31b4cb"
format_stamp: "Formatted at 2026-06-20 18:38:41 on dist-test-slave-d645"
server_key: "fba166657b7e62ec75111bb714148e1d"
server_key_iv: "8776d8c799cf8da9ef26c03a6674d24e"
server_key_version: "encryptionkey@0"
I20260620 18:38:41.964243 29816 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 18:38:41.967931 29831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:41.969274 29816 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:38:41.969429 29816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980676253068-24012-0/minicluster-data/ts-1/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980676253068-24012-0/minicluster-data/ts-1/wal
uuid: "026b703aea1d4408a40476b9ae31b4cb"
format_stamp: "Formatted at 2026-06-20 18:38:41 on dist-test-slave-d645"
server_key: "fba166657b7e62ec75111bb714148e1d"
server_key_iv: "8776d8c799cf8da9ef26c03a6674d24e"
server_key_version: "encryptionkey@0"
I20260620 18:38:41.969614 29816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980676253068-24012-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980676253068-24012-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980676253068-24012-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:41.997654 29816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:42.002086 29816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:42.002364 29816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:42.003401 29816 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:38:42.004848 29816 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:38:42.004926 29816 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:42.005049 29816 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:38:42.005100 29816 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:42.036713 29816 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.2:45539
I20260620 18:38:42.036761 29943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.2:45539 every 8 connection(s)
I20260620 18:38:42.037909 29816 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980676253068-24012-0/minicluster-data/ts-1/data/info.pb
I20260620 18:38:42.043762 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 29816
I20260620 18:38:42.043917 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980676253068-24012-0/minicluster-data/ts-1/wal/instance
I20260620 18:38:42.044320 24012 external_mini_cluster.cc:1428] Setting key d18b4c4f515448c65f3b319d3e3ea437
Jun 20 18:38:42 dist-test-slave-d645 krb5kdc[29576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980721, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.2@KRBTEST.COM for kudu/127.23.115.62@KRBTEST.COM
I20260620 18:38:42.057238 29812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:42.040511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.2:37167 (local address 127.23.115.62:33251)
0620 18:38:42.040740 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:42.040745 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:42.041823 (+ 1078us) server_negotiation.cc:407] Connection header received
0620 18:38:42.043654 (+ 1831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:42.043661 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:42.043740 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:42.043823 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:42.045344 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:42.046121 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:42.047647 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:42.048046 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:42.048155 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:42.051338 (+ 3183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:42.051342 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:42.051346 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:42.051377 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:42.054214 (+ 2837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:42.054945 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:42.054951 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:42.054955 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:42.055074 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:42.055473 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:42.055479 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:42.055482 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:42.055801 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:42.055949 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:42.056819 (+ 870us) server_negotiation.cc:299] Negotiation successful
0620 18:38:42.056996 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":49,"server-negotiator.queue_time_us":78}
I20260620 18:38:42.058610 29946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:42.040949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:33251 (local address 127.23.115.2:37167)
0620 18:38:42.041667 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:42.041705 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:42.043373 (+ 1668us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:42.044065 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:42.044074 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:42.044589 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:42.045189 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:42.045205 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:42.046266 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:42.046270 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:42.047513 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:42.047529 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:42.048251 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:42.048259 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:42.048471 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:42.049075 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:42.049097 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:42.051135 (+ 2038us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:42.054357 (+ 3222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:42.054367 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:42.054385 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:42.054805 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:42.055175 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:42.055180 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:42.055184 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:42.055362 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:42.055947 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:42.055956 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:42.056391 (+ 435us) client_negotiation.cc:769] Sending connection context
0620 18:38:42.056690 (+ 299us) client_negotiation.cc:240] Negotiation successful
0620 18:38:42.057197 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":287,"threads_started":1}
I20260620 18:38:42.059996 29944 heartbeater.cc:347] Connected to a master server at 127.23.115.62:33251
I20260620 18:38:42.060333 29944 heartbeater.cc:464] Registering TS with master...
I20260620 18:38:42.061221 29944 heartbeater.cc:511] Master 127.23.115.62:33251 requested a full tablet report, sending...
I20260620 18:38:42.062822 29621 ts_manager.cc:194] Registered new tserver with Master: 026b703aea1d4408a40476b9ae31b4cb (127.23.115.2:45539)
WARNING: no policy specified for kudu/127.23.115.3@KRBTEST.COM; defaulting to no policy
I20260620 18:38:42.063707 29621 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.115.2@KRBTEST.COM'} at 127.23.115.2:37167
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.115.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.115.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.3.keytab.
Entry for principal kudu/127.23.115.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.3.keytab.
I20260620 18:38:42.092037 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980676253068-24012-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980676253068-24012-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980676253068-24012-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980676253068-24012-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.115.3:0
--local_ip_for_outbound_sockets=127.23.115.3
--webserver_interface=127.23.115.3
--webserver_port=0
--tserver_master_addrs=127.23.115.62:33251
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.3.keytab
--principal=kudu/127.23.115.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:44121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:38:42.416911 29949 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:42.417210 29949 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:42.417327 29949 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:42.424630 29949 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:42.424738 29949 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:42.424881 29949 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.115.3
I20260620 18:38:42.434504 29949 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:44121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980676253068-24012-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980676253068-24012-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.3.keytab
--principal=kudu/127.23.115.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.115.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980676253068-24012-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.115.3
--webserver_port=0
--tserver_master_addrs=127.23.115.62:33251
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.115.3
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980676253068-24012-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:38:42.435760 29949 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:42.436983 29949 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:42.449625 29955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:42.449580 29954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:42.449550 29957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:42.450367 29949 server_base.cc:1034] running on GCE node
I20260620 18:38:42.450959 29949 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:42.451893 29949 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:42.453104 29949 hybrid_clock.cc:648] HybridClock initialized: now 1781980722453056 us; error 38 us; skew 500 ppm
I20260620 18:38:42.453419 29949 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:38:42 dist-test-slave-d645 krb5kdc[29576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980722, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:42.456843 29949 init.cc:377] Logged in from keytab as kudu/127.23.115.3@KRBTEST.COM (short username kudu)
I20260620 18:38:42.458269 29949 webserver.cc:460] Webserver started at http://127.23.115.3:41883/ using document root <none> and password file <none>
I20260620 18:38:42.458906 29949 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:42.459057 29949 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:42.459373 29949 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:38:42.462446 29949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980676253068-24012-0/minicluster-data/ts-2/data/instance:
uuid: "cbdaa0451d4c403da552e0e2fb178cb5"
format_stamp: "Formatted at 2026-06-20 18:38:42 on dist-test-slave-d645"
server_key: "43b17589f0aedbf6144d39925aa7e9fd"
server_key_iv: "421bc8f57b542fe444d746cbfe5873c5"
server_key_version: "encryptionkey@0"
I20260620 18:38:42.463261 29949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980676253068-24012-0/minicluster-data/ts-2/wal/instance:
uuid: "cbdaa0451d4c403da552e0e2fb178cb5"
format_stamp: "Formatted at 2026-06-20 18:38:42 on dist-test-slave-d645"
server_key: "43b17589f0aedbf6144d39925aa7e9fd"
server_key_iv: "421bc8f57b542fe444d746cbfe5873c5"
server_key_version: "encryptionkey@0"
I20260620 18:38:42.468636 29949 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 18:38:42.473353 29964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:42.474844 29949 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:38:42.475024 29949 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980676253068-24012-0/minicluster-data/ts-2/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980676253068-24012-0/minicluster-data/ts-2/wal
uuid: "cbdaa0451d4c403da552e0e2fb178cb5"
format_stamp: "Formatted at 2026-06-20 18:38:42 on dist-test-slave-d645"
server_key: "43b17589f0aedbf6144d39925aa7e9fd"
server_key_iv: "421bc8f57b542fe444d746cbfe5873c5"
server_key_version: "encryptionkey@0"
I20260620 18:38:42.475227 29949 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980676253068-24012-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980676253068-24012-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980676253068-24012-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:42.490538 29949 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:42.495299 29949 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:42.495559 29949 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:42.496484 29949 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:38:42.497756 29949 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:38:42.497831 29949 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:42.497961 29949 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:38:42.498013 29949 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:42.525090 29949 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.3:43561
I20260620 18:38:42.525169 30076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.3:43561 every 8 connection(s)
I20260620 18:38:42.526268 29949 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980676253068-24012-0/minicluster-data/ts-2/data/info.pb
I20260620 18:38:42.531800 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 29949
I20260620 18:38:42.531944 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980676253068-24012-0/minicluster-data/ts-2/wal/instance
I20260620 18:38:42.532351 24012 external_mini_cluster.cc:1428] Setting key 699b5fa3da84f1dc3e6713b8708dc3d7
Jun 20 18:38:42 dist-test-slave-d645 krb5kdc[29576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980722, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.3@KRBTEST.COM for kudu/127.23.115.62@KRBTEST.COM
I20260620 18:38:42.545418 29812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:42.528726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.3:50451 (local address 127.23.115.62:33251)
0620 18:38:42.528888 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:42.528893 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:42.530020 (+ 1127us) server_negotiation.cc:407] Connection header received
0620 18:38:42.531813 (+ 1793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:42.531818 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:42.531904 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:42.531992 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:42.533511 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:42.534286 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:42.535777 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:42.536164 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:42.536274 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:42.539623 (+ 3349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:42.539628 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:42.539631 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:42.539670 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:42.542426 (+ 2756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:42.543222 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:42.543226 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:42.543228 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:42.543356 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:42.543768 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:42.543772 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:42.543774 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:42.544033 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:42.544169 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:42.544824 (+ 655us) server_negotiation.cc:299] Negotiation successful
0620 18:38:42.545051 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 18:38:42.546409 30079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:42.529199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:33251 (local address 127.23.115.3:50451)
0620 18:38:42.529873 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:42.529907 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:42.531553 (+ 1646us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:42.532224 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:42.532232 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:42.532731 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:42.533356 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:42.533372 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:42.534426 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:42.534430 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:42.535644 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:42.535656 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:42.536360 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:42.536367 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:42.536580 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:42.537264 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:42.537285 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:42.539431 (+ 2146us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:42.542572 (+ 3141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:42.542581 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:42.542597 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:42.543077 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:42.543463 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:42.543468 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:42.543472 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:42.543651 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:42.544161 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:42.544168 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:42.544480 (+ 312us) client_negotiation.cc:769] Sending connection context
0620 18:38:42.544709 (+ 229us) client_negotiation.cc:240] Negotiation successful
0620 18:38:42.545208 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":272,"threads_started":1}
I20260620 18:38:42.547744 30077 heartbeater.cc:347] Connected to a master server at 127.23.115.62:33251
I20260620 18:38:42.548012 30077 heartbeater.cc:464] Registering TS with master...
I20260620 18:38:42.548722 30077 heartbeater.cc:511] Master 127.23.115.62:33251 requested a full tablet report, sending...
I20260620 18:38:42.550006 29620 ts_manager.cc:194] Registered new tserver with Master: cbdaa0451d4c403da552e0e2fb178cb5 (127.23.115.3:43561)
I20260620 18:38:42.550858 29620 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.115.3@KRBTEST.COM'} at 127.23.115.3:50451
I20260620 18:38:42.559708 24012 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:38:42.576207 29810 heartbeater.cc:503] Master 127.23.115.62:33251 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980676253068-24012-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980676253068-24012-0/krb5kdc/test-admin.keytab.
Jun 20 18:38:42 dist-test-slave-d645 krb5kdc[29576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980722, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:42.585187 24012 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260620 18:38:42.590706 30084 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:38:42.588150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43704 (local address 127.23.115.1:37791)
0620 18:38:42.588805 (+ 655us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:42.588918 (+ 113us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:42.588968 (+ 50us) server_negotiation.cc:407] Connection header received
0620 18:38:42.589049 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:42.589055 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:42.589265 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:42.589392 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:42.590358 (+ 966us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43704: BlockingRecv error: recv got EOF from 127.0.0.1:43704 (error 108)
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":297,"threads_started":1}
W20260620 18:38:42.590739 24012 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980676253068-24012-0/krb5kdc/krb5cc' of type FILE
Jun 20 18:38:42 dist-test-slave-d645 krb5kdc[29576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980722, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:42.592621 24012 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 18:38:42 dist-test-slave-d645 krb5kdc[29576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980722, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:42.593922 24012 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 18:38:42 dist-test-slave-d645 krb5kdc[29576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980722, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.115.1@KRBTEST.COM
I20260620 18:38:42.609941 30084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:42.595583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43712 (local address 127.23.115.1:37791)
0620 18:38:42.595772 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:42.595778 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:42.595935 (+ 157us) server_negotiation.cc:407] Connection header received
0620 18:38:42.596072 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:42.596076 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:42.596162 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:42.596245 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:42.597711 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:42.598663 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:42.599948 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:42.600524 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:42.600631 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:42.604667 (+ 4036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:42.604680 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:42.604695 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:42.604729 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:42.607332 (+ 2603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:42.607973 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:42.607980 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:42.607983 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:42.608111 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:42.608448 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:42.608452 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:42.608454 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:42.608891 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:42.609108 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:42.609408 (+ 300us) server_negotiation.cc:299] Negotiation successful
0620 18:38:42.609656 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260620 18:38:42.615672 30084 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:38:42.614005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43728 (local address 127.23.115.1:37791)
0620 18:38:42.614218 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:42.614223 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:42.614406 (+ 183us) server_negotiation.cc:407] Connection header received
0620 18:38:42.614561 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:42.614565 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:42.614632 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:42.614712 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:42.615480 (+ 768us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43728: BlockingRecv error: recv got EOF from 127.0.0.1:43728 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260620 18:38:42.616256 24012 init.cc:244] Error while opening credential cache '/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980676253068-24012-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980676253068-24012-0/krb5kdc/krb5cc)
Jun 20 18:38:42 dist-test-slave-d645 krb5kdc[29576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980722, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:42.617712 24012 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 18:38:42 dist-test-slave-d645 krb5kdc[29576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980722, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:42.618978 24012 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 18:38:42 dist-test-slave-d645 krb5kdc[29576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980722, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.115.1@KRBTEST.COM
I20260620 18:38:42.633598 30084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:42.620495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43730 (local address 127.23.115.1:37791)
0620 18:38:42.620717 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:42.620723 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:42.620992 (+ 269us) server_negotiation.cc:407] Connection header received
0620 18:38:42.621135 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:42.621140 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:42.621207 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:42.621289 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:42.622405 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:42.623179 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:42.624335 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:42.624759 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:42.624863 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:42.628200 (+ 3337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:42.628205 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:42.628208 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:42.628241 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:42.631028 (+ 2787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:42.631815 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:42.631820 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:42.631823 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:42.631970 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:42.632325 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:42.632330 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:42.632333 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:42.632601 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:42.632861 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:42.633128 (+ 267us) server_negotiation.cc:299] Negotiation successful
0620 18:38:42.633343 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260620 18:38:42.637954 30084 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:38:42.636195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43736 (local address 127.23.115.1:37791)
0620 18:38:42.636496 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:42.636503 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:42.636752 (+ 249us) server_negotiation.cc:407] Connection header received
0620 18:38:42.636889 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:42.636894 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:42.636974 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:42.637050 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:42.637740 (+ 690us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43736: BlockingRecv error: recv got EOF from 127.0.0.1:43736 (error 108)
Metrics: {"server-negotiator.queue_time_us":171}
W20260620 18:38:42.638280 24012 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980676253068-24012-0/krb5kdc/krb5cc' of type FILE
Jun 20 18:38:42 dist-test-slave-d645 krb5kdc[29576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980722, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:42.639847 24012 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 18:38:42 dist-test-slave-d645 krb5kdc[29576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980722, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:42.640980 24012 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 18:38:42 dist-test-slave-d645 krb5kdc[29576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980722, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.115.1@KRBTEST.COM
I20260620 18:38:42.656301 30084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:42.642512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43740 (local address 127.23.115.1:37791)
0620 18:38:42.642783 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:42.642790 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:42.643274 (+ 484us) server_negotiation.cc:407] Connection header received
0620 18:38:42.643575 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:42.643579 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:42.643642 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:42.643721 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:42.644849 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:42.645569 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:42.646998 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:42.647533 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:42.647685 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:42.651195 (+ 3510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:42.651201 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:42.651204 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:42.651247 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:42.653583 (+ 2336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:42.654207 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:42.654211 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:42.654214 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:42.654374 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:42.654854 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:42.654858 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:42.654860 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:42.655259 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:42.655500 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:42.655764 (+ 264us) server_negotiation.cc:299] Negotiation successful
0620 18:38:42.655942 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260620 18:38:42.658465 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 29682
I20260620 18:38:42.670670 30084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:42.659196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43750 (local address 127.23.115.1:37791)
0620 18:38:42.659345 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:42.659350 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:42.659670 (+ 320us) server_negotiation.cc:407] Connection header received
0620 18:38:42.659797 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:42.659800 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:42.659856 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:42.659932 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:42.660751 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:42.661461 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:42.662707 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:42.663217 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:42.663314 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:42.665636 (+ 2322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:42.665642 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:42.665645 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:42.665679 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:42.668223 (+ 2544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:42.668880 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:42.668884 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:42.668886 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:42.669010 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:42.669462 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:42.669467 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:42.669469 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:42.669737 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:42.669948 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:42.670251 (+ 303us) server_negotiation.cc:299] Negotiation successful
0620 18:38:42.670412 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 18:38:42.671301 29805 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:38:42.754279 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 29682
I20260620 18:38:42.767210 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 29816
Jun 20 18:38:42 dist-test-slave-d645 krb5kdc[29576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980722, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.115.2@KRBTEST.COM
I20260620 18:38:42.782312 30097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:42.768037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47000 (local address 127.23.115.2:45539)
0620 18:38:42.768567 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:42.768573 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:42.768598 (+ 25us) server_negotiation.cc:407] Connection header received
0620 18:38:42.768658 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:42.768665 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:42.768892 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:42.769013 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:42.769919 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:42.770923 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:42.772122 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:42.772716 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:42.772836 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:42.776614 (+ 3778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:42.776627 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:42.776641 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:42.776677 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:42.779569 (+ 2892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:42.780351 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:42.780358 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:42.780361 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:42.780488 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:42.780831 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:42.780834 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:42.780837 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:42.781190 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:42.781431 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:42.781788 (+ 357us) server_negotiation.cc:299] Negotiation successful
0620 18:38:42.781985 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":261,"threads_started":1}
I20260620 18:38:42.783697 29939 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:38:42.864540 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 29816
I20260620 18:38:42.876987 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 29949
Jun 20 18:38:42 dist-test-slave-d645 krb5kdc[29576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980722, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.115.3@KRBTEST.COM
I20260620 18:38:42.892527 30099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:42.878055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58466 (local address 127.23.115.3:43561)
0620 18:38:42.878742 (+ 687us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:42.878749 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:42.878780 (+ 31us) server_negotiation.cc:407] Connection header received
0620 18:38:42.878840 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:42.878846 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:42.879125 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:42.879298 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:42.880489 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:42.881519 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:42.882797 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:42.883408 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:42.883503 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:42.887336 (+ 3833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:42.887346 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:42.887361 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:42.887392 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:42.889960 (+ 2568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:42.890613 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:42.890620 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:42.890622 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:42.890736 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:42.891115 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:42.891121 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:42.891123 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:42.891437 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:42.891719 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:42.891980 (+ 261us) server_negotiation.cc:299] Negotiation successful
0620 18:38:42.892237 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":375,"threads_started":1}
I20260620 18:38:42.893777 30072 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:38:42.977102 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 29949
I20260620 18:38:42.990305 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 29589
I20260620 18:38:42.991407 29651 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:38:43.096009 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 29589
2026-06-20T18:38:43Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3785 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:38:43 dist-test-slave-d645 krb5kdc[30104](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:38:43 dist-test-slave-d645 krb5kdc[30104](info): set up 2 sockets
Jun 20 18:38:43 dist-test-slave-d645 krb5kdc[30104](info): commencing operation
krb5kdc: starting...
W20260620 18:38:44.123533 24012 mini_kdc.cc:121] Time spent starting KDC: real 0.988s user 0.008s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/krb5kdc/test-user.keytab.
Jun 20 18:38:44 dist-test-slave-d645 krb5kdc[30104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980724, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:38:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:38:44Z Disabled control of system clock
WARNING: no policy specified for oryx/127.23.115.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.115.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.115.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/krb5kdc/oryx_127.23.115.62.keytab.
Entry for principal oryx/127.23.115.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/krb5kdc/oryx_127.23.115.62.keytab.
I20260620 18:38:44.336140 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.115.62:36693
--webserver_interface=127.23.115.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/krb5kdc/oryx_127.23.115.62.keytab
--principal=oryx/127.23.115.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:34239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.115.62:36693
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:38:44.684687 30117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:44.684983 30117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:44.685115 30117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:44.692649 30117 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:38:44.692777 30117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:44.692838 30117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:44.692924 30117 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:38:44.692991 30117 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:38:44.704749 30117 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:34239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.115.62:36693
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/krb5kdc/oryx_127.23.115.62.keytab
--principal=oryx/127.23.115.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.115.62:36693
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.115.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:38:44.706369 30117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:44.708063 30117 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:44.720574 30122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:44.720729 30123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:44.720757 30125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:44.722272 30117 server_base.cc:1034] running on GCE node
I20260620 18:38:44.723300 30117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:44.724713 30117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:44.725947 30117 hybrid_clock.cc:648] HybridClock initialized: now 1781980724725882 us; error 46 us; skew 500 ppm
I20260620 18:38:44.726296 30117 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:38:44 dist-test-slave-d645 krb5kdc[30104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980724, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.115.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:44.729416 30117 init.cc:377] Logged in from keytab as oryx/127.23.115.62@KRBTEST.COM (short username oryx)
I20260620 18:38:44.730962 30117 webserver.cc:460] Webserver started at http://127.23.115.62:39301/ using document root <none> and password file <none>
I20260620 18:38:44.731627 30117 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:44.731745 30117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:44.732049 30117 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:38:44.735080 30117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/master-0/data/instance:
uuid: "8279b148b2454bfb90afa089240c8086"
format_stamp: "Formatted at 2026-06-20 18:38:44 on dist-test-slave-d645"
server_key: "78505ca477cba0774950d4546d8dcdb5"
server_key_iv: "888c08a443004ce1abaa661daafc9923"
server_key_version: "encryptionkey@0"
I20260620 18:38:44.735901 30117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/master-0/wal/instance:
uuid: "8279b148b2454bfb90afa089240c8086"
format_stamp: "Formatted at 2026-06-20 18:38:44 on dist-test-slave-d645"
server_key: "78505ca477cba0774950d4546d8dcdb5"
server_key_iv: "888c08a443004ce1abaa661daafc9923"
server_key_version: "encryptionkey@0"
I20260620 18:38:44.741075 30117 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:38:44.744735 30132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:44.746137 30117 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 18:38:44.746335 30117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/master-0/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/master-0/wal
uuid: "8279b148b2454bfb90afa089240c8086"
format_stamp: "Formatted at 2026-06-20 18:38:44 on dist-test-slave-d645"
server_key: "78505ca477cba0774950d4546d8dcdb5"
server_key_iv: "888c08a443004ce1abaa661daafc9923"
server_key_version: "encryptionkey@0"
I20260620 18:38:44.746515 30117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:44.778750 30117 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:44.783638 30117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:44.783912 30117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:44.801829 30117 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.62:36693
I20260620 18:38:44.801863 30183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.62:36693 every 8 connection(s)
I20260620 18:38:44.803181 30117 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
I20260620 18:38:44.806726 30184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:44.807325 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 30117
I20260620 18:38:44.807526 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/master-0/wal/instance
I20260620 18:38:44.808183 24012 external_mini_cluster.cc:1428] Setting key 527a768e5de18a5d637afe7e47a7e79f
I20260620 18:38:44.819859 30184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8279b148b2454bfb90afa089240c8086: Bootstrap starting.
Jun 20 18:38:44 dist-test-slave-d645 krb5kdc[30104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980724, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM
I20260620 18:38:44.823424 30184 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8279b148b2454bfb90afa089240c8086: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:44.824751 30184 log.cc:824] T 00000000000000000000000000000000 P 8279b148b2454bfb90afa089240c8086: Log is configured to *not* fsync() on all Append() calls
I20260620 18:38:44.828296 30184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8279b148b2454bfb90afa089240c8086: No bootstrap required, opened a new log
I20260620 18:38:44.832346 30187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:44.810030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53546 (local address 127.23.115.62:36693)
0620 18:38:44.810767 (+ 737us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:44.810781 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:44.810821 (+ 40us) server_negotiation.cc:407] Connection header received
0620 18:38:44.812467 (+ 1646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:44.812488 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:44.812865 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:44.813190 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:44.814825 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:44.816132 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:44.817836 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:44.818664 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:44.818830 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:44.823592 (+ 4762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:44.823605 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:44.823627 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:44.823678 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:44.827472 (+ 3794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:44.828179 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:44.828187 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:44.828196 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:44.828424 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:44.828836 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:44.828841 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:44.828845 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:44.829541 (+ 696us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:44.829963 (+ 422us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:44.830265 (+ 302us) server_negotiation.cc:299] Negotiation successful
0620 18:38:44.830898 (+ 633us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":318,"threads_started":1}
I20260620 18:38:44.837754 30184 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8279b148b2454bfb90afa089240c8086 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8279b148b2454bfb90afa089240c8086" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 36693 } }
I20260620 18:38:44.838120 30184 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8279b148b2454bfb90afa089240c8086 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:44.838225 30184 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8279b148b2454bfb90afa089240c8086 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8279b148b2454bfb90afa089240c8086, State: Initialized, Role: FOLLOWER
I20260620 18:38:44.838793 30184 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8279b148b2454bfb90afa089240c8086 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8279b148b2454bfb90afa089240c8086" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 36693 } }
I20260620 18:38:44.839051 30184 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8279b148b2454bfb90afa089240c8086 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:38:44.839179 30184 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8279b148b2454bfb90afa089240c8086 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:38:44.839324 30184 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8279b148b2454bfb90afa089240c8086 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:44.841368 30184 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8279b148b2454bfb90afa089240c8086 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8279b148b2454bfb90afa089240c8086" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 36693 } }
I20260620 18:38:44.841959 30184 leader_election.cc:302] T 00000000000000000000000000000000 P 8279b148b2454bfb90afa089240c8086 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8279b148b2454bfb90afa089240c8086; no voters:
I20260620 18:38:44.842650 30184 leader_election.cc:288] T 00000000000000000000000000000000 P 8279b148b2454bfb90afa089240c8086 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:38:44.842787 30189 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8279b148b2454bfb90afa089240c8086 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:38:44.843403 30189 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8279b148b2454bfb90afa089240c8086 [term 1 LEADER]: Becoming Leader. State: Replica: 8279b148b2454bfb90afa089240c8086, State: Running, Role: LEADER
I20260620 18:38:44.843917 30189 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8279b148b2454bfb90afa089240c8086 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8279b148b2454bfb90afa089240c8086" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 36693 } }
I20260620 18:38:44.844674 30184 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8279b148b2454bfb90afa089240c8086 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:38:44.846458 30190 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8279b148b2454bfb90afa089240c8086 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8279b148b2454bfb90afa089240c8086" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8279b148b2454bfb90afa089240c8086" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 36693 } } }
I20260620 18:38:44.846846 30190 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8279b148b2454bfb90afa089240c8086 [sys.catalog]: This master's current role is: LEADER
I20260620 18:38:44.847918 30191 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8279b148b2454bfb90afa089240c8086 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8279b148b2454bfb90afa089240c8086. Latest consensus state: current_term: 1 leader_uuid: "8279b148b2454bfb90afa089240c8086" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8279b148b2454bfb90afa089240c8086" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 36693 } } }
I20260620 18:38:44.848214 30191 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8279b148b2454bfb90afa089240c8086 [sys.catalog]: This master's current role is: LEADER
I20260620 18:38:44.848881 30196 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:38:44.853041 30196 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:38:44.860620 30196 catalog_manager.cc:1347] Generated new cluster ID: f0357000094b4f94b558a6c421dae89d
I20260620 18:38:44.860720 30196 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:38:44.892235 30196 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:38:44.893567 30196 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:38:44.916622 30196 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8279b148b2454bfb90afa089240c8086: Generated new TSK 0
I20260620 18:38:44.917418 30196 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.23.115.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.115.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.115.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/krb5kdc/oryx_127.23.115.1.keytab.
Entry for principal oryx/127.23.115.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/krb5kdc/oryx_127.23.115.1.keytab.
I20260620 18:38:44.961673 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.115.1:0
--local_ip_for_outbound_sockets=127.23.115.1
--webserver_interface=127.23.115.1
--webserver_port=0
--tserver_master_addrs=127.23.115.62:36693
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/krb5kdc/oryx_127.23.115.1.keytab
--principal=oryx/127.23.115.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:34239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:38:45.294109 30210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:45.294400 30210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:45.294513 30210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:45.302053 30210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:45.302197 30210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:45.302359 30210 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.115.1
I20260620 18:38:45.313259 30210 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:34239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/krb5kdc/oryx_127.23.115.1.keytab
--principal=oryx/127.23.115.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.115.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.115.1
--webserver_port=0
--tserver_master_addrs=127.23.115.62:36693
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.115.1
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:38:45.314569 30210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:45.315901 30210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:45.328971 30215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:45.328930 30216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:45.328887 30218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:45.330160 30210 server_base.cc:1034] running on GCE node
I20260620 18:38:45.330791 30210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:45.331808 30210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:45.332978 30210 hybrid_clock.cc:648] HybridClock initialized: now 1781980725332941 us; error 32 us; skew 500 ppm
I20260620 18:38:45.333302 30210 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:38:45 dist-test-slave-d645 krb5kdc[30104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980725, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.115.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:45.336807 30210 init.cc:377] Logged in from keytab as oryx/127.23.115.1@KRBTEST.COM (short username oryx)
I20260620 18:38:45.338280 30210 webserver.cc:460] Webserver started at http://127.23.115.1:43589/ using document root <none> and password file <none>
I20260620 18:38:45.338992 30210 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:45.339112 30210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:45.339411 30210 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:38:45.342299 30210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/ts-0/data/instance:
uuid: "74598bd1acf5422181769e91580ab45a"
format_stamp: "Formatted at 2026-06-20 18:38:45 on dist-test-slave-d645"
server_key: "a69e9c68c12ca5a0ad169079ef965306"
server_key_iv: "962521789ee1c99438cfbc637e7273c4"
server_key_version: "encryptionkey@0"
I20260620 18:38:45.343290 30210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/ts-0/wal/instance:
uuid: "74598bd1acf5422181769e91580ab45a"
format_stamp: "Formatted at 2026-06-20 18:38:45 on dist-test-slave-d645"
server_key: "a69e9c68c12ca5a0ad169079ef965306"
server_key_iv: "962521789ee1c99438cfbc637e7273c4"
server_key_version: "encryptionkey@0"
I20260620 18:38:45.349915 30210 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260620 18:38:45.353982 30225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:45.355470 30210 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:38:45.355633 30210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/ts-0/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/ts-0/wal
uuid: "74598bd1acf5422181769e91580ab45a"
format_stamp: "Formatted at 2026-06-20 18:38:45 on dist-test-slave-d645"
server_key: "a69e9c68c12ca5a0ad169079ef965306"
server_key_iv: "962521789ee1c99438cfbc637e7273c4"
server_key_version: "encryptionkey@0"
I20260620 18:38:45.355859 30210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:45.398877 30210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:45.403741 30210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:45.404119 30210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:45.405095 30210 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:38:45.406423 30210 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:38:45.406495 30210 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:45.406584 30210 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:38:45.406626 30210 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:45.440802 30210 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.1:39357
I20260620 18:38:45.440852 30337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.1:39357 every 8 connection(s)
I20260620 18:38:45.441996 30210 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/ts-0/data/info.pb
I20260620 18:38:45.442674 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 30210
I20260620 18:38:45.443115 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/ts-0/wal/instance
I20260620 18:38:45.444008 24012 external_mini_cluster.cc:1428] Setting key 8cb4b642eb068f8a873cba53c5bc792c
Jun 20 18:38:45 dist-test-slave-d645 krb5kdc[30104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980725, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.115.1@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.23.115.2@KRBTEST.COM; defaulting to no policy
I20260620 18:38:45.463452 30340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:45.445024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.1:59291 (local address 127.23.115.62:36693)
0620 18:38:45.446334 (+ 1310us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:45.446340 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:45.447162 (+ 822us) server_negotiation.cc:407] Connection header received
0620 18:38:45.448948 (+ 1786us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:45.448953 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:45.449034 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:45.449157 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:45.450725 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:45.451506 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:45.453011 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:45.453408 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:45.453519 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:45.457296 (+ 3777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:45.457301 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:45.457304 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:45.457338 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:45.460397 (+ 3059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:45.461183 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:45.461189 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:45.461191 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:45.461307 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:45.461712 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:45.461717 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:45.461721 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:45.461990 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:45.462148 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:45.462849 (+ 701us) server_negotiation.cc:299] Negotiation successful
0620 18:38:45.463104 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1184,"thread_start_us":1053,"threads_started":1}
I20260620 18:38:45.464387 30341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:45.446220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:36693 (local address 127.23.115.1:59291)
0620 18:38:45.447020 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:45.447065 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:45.448692 (+ 1627us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:45.449411 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:45.449420 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:45.449982 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:45.450562 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:45.450581 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:45.451671 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:45.451675 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:45.452868 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:45.452884 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:45.453631 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:45.453639 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:45.453842 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:45.454547 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:45.454570 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:45.457066 (+ 2496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:45.460528 (+ 3462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:45.460537 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:45.460573 (+ 36us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:45.461066 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:45.461408 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:45.461413 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:45.461416 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:45.461587 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:45.462116 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:45.462125 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:45.462454 (+ 329us) client_negotiation.cc:769] Sending connection context
0620 18:38:45.462735 (+ 281us) client_negotiation.cc:240] Negotiation successful
0620 18:38:45.463285 (+ 550us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":335,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.115.2@KRBTEST.COM" created.
I20260620 18:38:45.466161 30338 heartbeater.cc:347] Connected to a master server at 127.23.115.62:36693
I20260620 18:38:45.466495 30338 heartbeater.cc:464] Registering TS with master...
I20260620 18:38:45.467312 30338 heartbeater.cc:511] Master 127.23.115.62:36693 requested a full tablet report, sending...
I20260620 18:38:45.469580 30149 ts_manager.cc:194] Registered new tserver with Master: 74598bd1acf5422181769e91580ab45a (127.23.115.1:39357)
I20260620 18:38:45.471465 30149 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.115.1@KRBTEST.COM'} at 127.23.115.1:59291
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.115.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/krb5kdc/oryx_127.23.115.2.keytab.
Entry for principal oryx/127.23.115.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/krb5kdc/oryx_127.23.115.2.keytab.
I20260620 18:38:45.490437 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.115.2:0
--local_ip_for_outbound_sockets=127.23.115.2
--webserver_interface=127.23.115.2
--webserver_port=0
--tserver_master_addrs=127.23.115.62:36693
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/krb5kdc/oryx_127.23.115.2.keytab
--principal=oryx/127.23.115.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:34239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:38:45.801059 30344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:45.801349 30344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:45.801496 30344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:45.809326 30344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:45.809441 30344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:45.809597 30344 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.115.2
I20260620 18:38:45.819290 30344 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:34239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/krb5kdc/oryx_127.23.115.2.keytab
--principal=oryx/127.23.115.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.115.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.115.2
--webserver_port=0
--tserver_master_addrs=127.23.115.62:36693
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.115.2
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:38:45.820487 30344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:45.821682 30344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:45.834208 30349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:45.834208 30352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:45.834290 30350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:45.835019 30344 server_base.cc:1034] running on GCE node
I20260620 18:38:45.835592 30344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:45.836460 30344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:45.837651 30344 hybrid_clock.cc:648] HybridClock initialized: now 1781980725837610 us; error 33 us; skew 500 ppm
I20260620 18:38:45.837935 30344 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:38:45 dist-test-slave-d645 krb5kdc[30104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980725, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.115.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:45.840940 30344 init.cc:377] Logged in from keytab as oryx/127.23.115.2@KRBTEST.COM (short username oryx)
I20260620 18:38:45.842522 30344 webserver.cc:460] Webserver started at http://127.23.115.2:36175/ using document root <none> and password file <none>
I20260620 18:38:45.843283 30344 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:45.843400 30344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:45.843683 30344 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:38:45.846926 30344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/ts-1/data/instance:
uuid: "7b659e72cc154daf8c7b6de5cc3d0c29"
format_stamp: "Formatted at 2026-06-20 18:38:45 on dist-test-slave-d645"
server_key: "7e516ef6f859822c2c3689cf97877175"
server_key_iv: "723d0e5857a4112414f630619f06e2a3"
server_key_version: "encryptionkey@0"
I20260620 18:38:45.847949 30344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/ts-1/wal/instance:
uuid: "7b659e72cc154daf8c7b6de5cc3d0c29"
format_stamp: "Formatted at 2026-06-20 18:38:45 on dist-test-slave-d645"
server_key: "7e516ef6f859822c2c3689cf97877175"
server_key_iv: "723d0e5857a4112414f630619f06e2a3"
server_key_version: "encryptionkey@0"
I20260620 18:38:45.853528 30344 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260620 18:38:45.857220 30359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:45.858593 30344 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:38:45.858740 30344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/ts-1/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/ts-1/wal
uuid: "7b659e72cc154daf8c7b6de5cc3d0c29"
format_stamp: "Formatted at 2026-06-20 18:38:45 on dist-test-slave-d645"
server_key: "7e516ef6f859822c2c3689cf97877175"
server_key_iv: "723d0e5857a4112414f630619f06e2a3"
server_key_version: "encryptionkey@0"
I20260620 18:38:45.858922 30344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:45.882442 30344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:45.886523 30344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:45.886768 30344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:45.887761 30344 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:38:45.889066 30344 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:38:45.889137 30344 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:45.889225 30344 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:38:45.889268 30344 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:45.920151 30344 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.2:33167
I20260620 18:38:45.920200 30471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.2:33167 every 8 connection(s)
I20260620 18:38:45.921382 30344 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/ts-1/data/info.pb
I20260620 18:38:45.929329 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 30344
I20260620 18:38:45.929522 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/ts-1/wal/instance
I20260620 18:38:45.929980 24012 external_mini_cluster.cc:1428] Setting key 547b44dcd273a806061ca3e5bdad5b5f
Jun 20 18:38:45 dist-test-slave-d645 krb5kdc[30104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980725, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.115.2@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM
I20260620 18:38:45.943634 30340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:45.924199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.2:41403 (local address 127.23.115.62:36693)
0620 18:38:45.924461 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:45.924469 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:45.925744 (+ 1275us) server_negotiation.cc:407] Connection header received
0620 18:38:45.927643 (+ 1899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:45.927648 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:45.927737 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:45.927833 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:45.929620 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:45.930375 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:45.932616 (+ 2241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:45.933044 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:45.933141 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:45.937763 (+ 4622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:45.937771 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:45.937777 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:45.937826 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:45.940625 (+ 2799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:45.941355 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:45.941359 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:45.941361 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:45.941488 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:45.941894 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:45.941898 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:45.941900 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:45.942172 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:45.942301 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:45.943062 (+ 761us) server_negotiation.cc:299] Negotiation successful
0620 18:38:45.943300 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260620 18:38:45.944686 30474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:45.924681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:36693 (local address 127.23.115.2:41403)
0620 18:38:45.925545 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:45.925588 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:45.927338 (+ 1750us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:45.928084 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:45.928093 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:45.928622 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:45.929408 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:45.929444 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:45.930522 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:45.930526 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:45.932443 (+ 1917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:45.932463 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:45.933296 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:45.933304 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:45.933637 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:45.934428 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:45.934460 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:45.937446 (+ 2986us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:45.940781 (+ 3335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:45.940788 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:45.940803 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:45.941227 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:45.941594 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:45.941598 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:45.941600 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:45.941780 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:45.942290 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:45.942299 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:45.942646 (+ 347us) client_negotiation.cc:769] Sending connection context
0620 18:38:45.942891 (+ 245us) client_negotiation.cc:240] Negotiation successful
0620 18:38:45.943513 (+ 622us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":310,"threads_started":1}
I20260620 18:38:45.946420 30472 heartbeater.cc:347] Connected to a master server at 127.23.115.62:36693
I20260620 18:38:45.946729 30472 heartbeater.cc:464] Registering TS with master...
I20260620 18:38:45.947465 30472 heartbeater.cc:511] Master 127.23.115.62:36693 requested a full tablet report, sending...
I20260620 18:38:45.948951 30149 ts_manager.cc:194] Registered new tserver with Master: 7b659e72cc154daf8c7b6de5cc3d0c29 (127.23.115.2:33167)
WARNING: no policy specified for oryx/127.23.115.3@KRBTEST.COM; defaulting to no policy
I20260620 18:38:45.949792 30149 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.115.2@KRBTEST.COM'} at 127.23.115.2:41403
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.115.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.115.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/krb5kdc/oryx_127.23.115.3.keytab.
Entry for principal oryx/127.23.115.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/krb5kdc/oryx_127.23.115.3.keytab.
I20260620 18:38:45.974435 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.115.3:0
--local_ip_for_outbound_sockets=127.23.115.3
--webserver_interface=127.23.115.3
--webserver_port=0
--tserver_master_addrs=127.23.115.62:36693
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/krb5kdc/oryx_127.23.115.3.keytab
--principal=oryx/127.23.115.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:34239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:38:46.298518 30477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:46.298810 30477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:46.298905 30477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:46.306241 30477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:46.306351 30477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:46.306471 30477 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.115.3
I20260620 18:38:46.317176 30477 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:34239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/krb5kdc/oryx_127.23.115.3.keytab
--principal=oryx/127.23.115.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.115.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.115.3
--webserver_port=0
--tserver_master_addrs=127.23.115.62:36693
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.115.3
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:38:46.318387 30477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:46.319628 30477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:46.331980 30482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:46.331998 30483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:46.332279 30485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:46.332902 30477 server_base.cc:1034] running on GCE node
I20260620 18:38:46.333529 30477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:46.334554 30477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:46.335784 30477 hybrid_clock.cc:648] HybridClock initialized: now 1781980726335729 us; error 41 us; skew 500 ppm
I20260620 18:38:46.336122 30477 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:38:46 dist-test-slave-d645 krb5kdc[30104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980726, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.115.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:46.339234 30477 init.cc:377] Logged in from keytab as oryx/127.23.115.3@KRBTEST.COM (short username oryx)
I20260620 18:38:46.340641 30477 webserver.cc:460] Webserver started at http://127.23.115.3:35111/ using document root <none> and password file <none>
I20260620 18:38:46.341279 30477 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:46.341377 30477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:46.341661 30477 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:38:46.344512 30477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/ts-2/data/instance:
uuid: "b8959fed0d23477fab84fd0a790447ce"
format_stamp: "Formatted at 2026-06-20 18:38:46 on dist-test-slave-d645"
server_key: "3d4b70bd1a5b2a78f305ccd161da052b"
server_key_iv: "4c8563e2882971ba9d9c39d82aee57e5"
server_key_version: "encryptionkey@0"
I20260620 18:38:46.345361 30477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/ts-2/wal/instance:
uuid: "b8959fed0d23477fab84fd0a790447ce"
format_stamp: "Formatted at 2026-06-20 18:38:46 on dist-test-slave-d645"
server_key: "3d4b70bd1a5b2a78f305ccd161da052b"
server_key_iv: "4c8563e2882971ba9d9c39d82aee57e5"
server_key_version: "encryptionkey@0"
I20260620 18:38:46.350502 30477 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 18:38:46.354157 30492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:46.355650 30477 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 18:38:46.355801 30477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/ts-2/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/ts-2/wal
uuid: "b8959fed0d23477fab84fd0a790447ce"
format_stamp: "Formatted at 2026-06-20 18:38:46 on dist-test-slave-d645"
server_key: "3d4b70bd1a5b2a78f305ccd161da052b"
server_key_iv: "4c8563e2882971ba9d9c39d82aee57e5"
server_key_version: "encryptionkey@0"
I20260620 18:38:46.355986 30477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:46.371134 30477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:46.375119 30477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:46.375399 30477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:46.376389 30477 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:38:46.377693 30477 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:38:46.377763 30477 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:46.377867 30477 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:38:46.377923 30477 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:46.409307 30477 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.3:44857
I20260620 18:38:46.409360 30604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.3:44857 every 8 connection(s)
I20260620 18:38:46.410523 30477 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/ts-2/data/info.pb
I20260620 18:38:46.416152 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 30477
I20260620 18:38:46.416390 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/ts-2/wal/instance
I20260620 18:38:46.416947 24012 external_mini_cluster.cc:1428] Setting key 17615a9730710052d92fe6fb4bf02f01
Jun 20 18:38:46 dist-test-slave-d645 krb5kdc[30104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980726, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.115.3@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM
I20260620 18:38:46.429711 30340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:46.412988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.3:47261 (local address 127.23.115.62:36693)
0620 18:38:46.413217 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:46.413223 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:46.414347 (+ 1124us) server_negotiation.cc:407] Connection header received
0620 18:38:46.416230 (+ 1883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:46.416236 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:46.416333 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:46.416428 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:46.418114 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:46.419033 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:46.420498 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:46.420905 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:46.421012 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:46.424263 (+ 3251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:46.424268 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:46.424271 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:46.424302 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:46.426869 (+ 2567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:46.427562 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:46.427566 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:46.427569 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:46.427695 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:46.428086 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:46.428090 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:46.428092 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:46.428361 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:46.428490 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:46.429170 (+ 680us) server_negotiation.cc:299] Negotiation successful
0620 18:38:46.429382 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260620 18:38:46.430655 30607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:46.413448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:36693 (local address 127.23.115.3:47261)
0620 18:38:46.414189 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:46.414231 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:46.415922 (+ 1691us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:46.416697 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:46.416708 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:46.417400 (+ 692us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:46.417952 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:46.417969 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:46.419187 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:46.419191 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:46.420362 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:46.420374 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:46.421111 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:46.421118 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:46.421338 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:46.421948 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:46.421970 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:46.424069 (+ 2099us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:46.427021 (+ 2952us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:46.427030 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:46.427044 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:46.427452 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:46.427794 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:46.427799 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:46.427803 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:46.427976 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:46.428482 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:46.428489 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:46.428806 (+ 317us) client_negotiation.cc:769] Sending connection context
0620 18:38:46.429034 (+ 228us) client_negotiation.cc:240] Negotiation successful
0620 18:38:46.429523 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":270,"threads_started":1}
I20260620 18:38:46.432134 30605 heartbeater.cc:347] Connected to a master server at 127.23.115.62:36693
I20260620 18:38:46.432443 30605 heartbeater.cc:464] Registering TS with master...
I20260620 18:38:46.433235 30605 heartbeater.cc:511] Master 127.23.115.62:36693 requested a full tablet report, sending...
I20260620 18:38:46.434499 30149 ts_manager.cc:194] Registered new tserver with Master: b8959fed0d23477fab84fd0a790447ce (127.23.115.3:44857)
I20260620 18:38:46.435374 30149 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.115.3@KRBTEST.COM'} at 127.23.115.3:47261
I20260620 18:38:46.445786 24012 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.23.115.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.115.61@KRBTEST.COM" created.
I20260620 18:38:46.475234 30338 heartbeater.cc:503] Master 127.23.115.62:36693 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.115.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/krb5kdc/oryx_127.23.115.61.keytab.
Entry for principal oryx/127.23.115.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/krb5kdc/oryx_127.23.115.61.keytab.
I20260620 18:38:46.502789 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.115.61:45651
--webserver_interface=127.23.115.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/krb5kdc/oryx_127.23.115.61.keytab
--principal=oryx/127.23.115.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:34239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.115.62:36693,127.23.115.61:45651
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:38:46.858616 30610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:46.858908 30610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:46.859067 30610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:46.866516 30610 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:38:46.866621 30610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:46.866669 30610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:46.866724 30610 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:38:46.866765 30610 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:38:46.876605 30610 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:34239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.23.115.62:36693,127.23.115.61:45651
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/krb5kdc/oryx_127.23.115.61.keytab
--principal=oryx/127.23.115.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.115.61:45651
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.23.115.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
W20260620 18:38:46.876917 30610 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.23.115.62:36693,127.23.115.61:45651'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260620 18:38:46.877919 30610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:46.879207 30610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:46.890816 30615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:46.890755 30616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:46.890758 30618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:46.891526 30610 server_base.cc:1034] running on GCE node
I20260620 18:38:46.892297 30610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:46.893662 30610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:46.894881 30610 hybrid_clock.cc:648] HybridClock initialized: now 1781980726894808 us; error 53 us; skew 500 ppm
I20260620 18:38:46.895217 30610 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:38:46 dist-test-slave-d645 krb5kdc[30104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980726, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.115.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:46.898196 30610 init.cc:377] Logged in from keytab as oryx/127.23.115.61@KRBTEST.COM (short username oryx)
I20260620 18:38:46.899680 30610 webserver.cc:460] Webserver started at http://127.23.115.61:32929/ using document root <none> and password file <none>
I20260620 18:38:46.900352 30610 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:46.900449 30610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:46.900748 30610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:38:46.903831 30610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/master-1/data/instance:
uuid: "0d00dab869714576ae199f6caf0c39d8"
format_stamp: "Formatted at 2026-06-20 18:38:46 on dist-test-slave-d645"
server_key: "1dc7c1c901e9941b4a685355d7cea31e"
server_key_iv: "0f1bc00141961b90ce7ee742b1ea39aa"
server_key_version: "encryptionkey@0"
I20260620 18:38:46.904575 30610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/master-1/wal/instance:
uuid: "0d00dab869714576ae199f6caf0c39d8"
format_stamp: "Formatted at 2026-06-20 18:38:46 on dist-test-slave-d645"
server_key: "1dc7c1c901e9941b4a685355d7cea31e"
server_key_iv: "0f1bc00141961b90ce7ee742b1ea39aa"
server_key_version: "encryptionkey@0"
I20260620 18:38:46.909603 30610 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 18:38:46.912997 30625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:46.914345 30610 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:38:46.914503 30610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/master-1/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/master-1/wal
uuid: "0d00dab869714576ae199f6caf0c39d8"
format_stamp: "Formatted at 2026-06-20 18:38:46 on dist-test-slave-d645"
server_key: "1dc7c1c901e9941b4a685355d7cea31e"
server_key_iv: "0f1bc00141961b90ce7ee742b1ea39aa"
server_key_version: "encryptionkey@0"
I20260620 18:38:46.914711 30610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:46.929363 30610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:46.933516 30610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:46.933761 30610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:46.949517 30610 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.61:45651
I20260620 18:38:46.949553 30676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.61:45651 every 8 connection(s)
I20260620 18:38:46.950863 30610 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/master-1/data/info.pb
I20260620 18:38:46.952512 30472 heartbeater.cc:503] Master 127.23.115.62:36693 was elected leader, sending a full tablet report...
I20260620 18:38:46.954883 30677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:46.955278 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 30610
I20260620 18:38:46.964368 30677 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.115.62" port: 36693 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:38:46 dist-test-slave-d645 krb5kdc[30104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980724, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.115.61@KRBTEST.COM
Jun 20 18:38:46 dist-test-slave-d645 krb5kdc[30104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980726, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.115.61@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM
I20260620 18:38:46.980996 30681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:46.967098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53562 (local address 127.23.115.62:36693)
0620 18:38:46.967524 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:46.967528 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:46.967972 (+ 444us) server_negotiation.cc:407] Connection header received
0620 18:38:46.968182 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:46.968186 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:46.968267 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:46.968361 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:46.969413 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:46.970092 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:46.971461 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:46.972075 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:46.972267 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:46.975549 (+ 3282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:46.975554 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:46.975557 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:46.975589 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:46.978337 (+ 2748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:46.979026 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:46.979030 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:46.979032 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:46.979157 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:46.979507 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:46.979510 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:46.979512 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:46.979808 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:46.979936 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:46.980415 (+ 479us) server_negotiation.cc:299] Negotiation successful
0620 18:38:46.980642 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":215,"threads_started":1}
I20260620 18:38:46.981899 30682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:46.967108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:36693 (local address 127.0.0.1:53562)
0620 18:38:46.967844 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:46.967883 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:46.968039 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:46.968459 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:46.968464 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:46.969024 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:46.969263 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:46.969272 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:46.970221 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:46.970225 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:46.971320 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:46.971336 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:46.972345 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:46.972350 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:46.972477 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:46.973066 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:46.973087 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:46.975327 (+ 2240us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:46.978454 (+ 3127us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:46.978465 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:46.978473 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:46.978891 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:46.979260 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:46.979265 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:46.979269 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:46.979401 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:46.979936 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:46.979944 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:46.980128 (+ 184us) client_negotiation.cc:769] Sending connection context
0620 18:38:46.980289 (+ 161us) client_negotiation.cc:240] Negotiation successful
0620 18:38:46.980698 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":370,"threads_started":1}
I20260620 18:38:46.981915 30680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:46.957621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45590 (local address 127.23.115.61:45651)
0620 18:38:46.958485 (+ 864us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:46.958496 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:46.958542 (+ 46us) server_negotiation.cc:407] Connection header received
0620 18:38:46.960243 (+ 1701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:46.960265 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:46.960652 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:46.961015 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:46.962424 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:46.963823 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:46.965576 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:46.966214 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:46.966334 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:46.974599 (+ 8265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:46.974609 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:46.974633 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:46.974673 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:46.977541 (+ 2868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:46.978352 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:46.978359 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:46.978365 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:46.978578 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:46.978991 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:46.978996 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:46.978999 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:46.979521 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:46.979778 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:46.980226 (+ 448us) server_negotiation.cc:299] Negotiation successful
0620 18:38:46.980826 (+ 600us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":110,"server-negotiator.queue_time_us":672,"spinlock_wait_cycles":15104,"thread_start_us":440,"threads_started":1}
I20260620 18:38:46.983743 30677 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.115.61" port: 45651 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:38:46 dist-test-slave-d645 krb5kdc[30104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980726, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.115.61@KRBTEST.COM for oryx/127.23.115.61@KRBTEST.COM
I20260620 18:38:46.995402 30682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:46.984380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.61:45651 (local address 127.0.0.1:45592)
0620 18:38:46.984572 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:46.984591 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:46.984678 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:46.984977 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:46.984980 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:46.985309 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:46.985441 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:46.985449 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:46.986413 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:46.986416 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:46.987342 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:46.987354 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:46.987977 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:46.987982 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:46.988124 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:46.988720 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:46.988737 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:46.990660 (+ 1923us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:46.993308 (+ 2648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:46.993312 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:46.993315 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:46.993690 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:46.993992 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:46.993996 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:46.993998 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:46.994153 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:46.994625 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:46.994628 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:46.994757 (+ 129us) client_negotiation.cc:769] Sending connection context
0620 18:38:46.994867 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 18:38:46.995149 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260620 18:38:46.995607 30680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:46.984626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45592 (local address 127.23.115.61:45651)
0620 18:38:46.984771 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:46.984775 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:46.984789 (+ 14us) server_negotiation.cc:407] Connection header received
0620 18:38:46.984824 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:46.984827 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:46.984874 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:46.984944 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:46.985564 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:46.986285 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:46.987462 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:46.987833 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:46.987929 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:46.990826 (+ 2897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:46.990831 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:46.990834 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:46.990862 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:46.993173 (+ 2311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:46.993794 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:46.993798 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:46.993800 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:46.993903 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:46.994259 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:46.994262 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:46.994264 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:46.994517 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:46.994619 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:46.994985 (+ 366us) server_negotiation.cc:299] Negotiation successful
0620 18:38:46.995312 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":125,"server-negotiator.queue_time_us":52}
I20260620 18:38:47.000209 30677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d00dab869714576ae199f6caf0c39d8: Bootstrap starting.
I20260620 18:38:47.003562 30677 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d00dab869714576ae199f6caf0c39d8: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:47.004706 30677 log.cc:824] T 00000000000000000000000000000000 P 0d00dab869714576ae199f6caf0c39d8: Log is configured to *not* fsync() on all Append() calls
I20260620 18:38:47.008910 30677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d00dab869714576ae199f6caf0c39d8: No bootstrap required, opened a new log
I20260620 18:38:47.016687 30677 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d00dab869714576ae199f6caf0c39d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8279b148b2454bfb90afa089240c8086" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 36693 } } peers { permanent_uuid: "0d00dab869714576ae199f6caf0c39d8" member_type: VOTER last_known_addr { host: "127.23.115.61" port: 45651 } }
I20260620 18:38:47.017096 30677 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d00dab869714576ae199f6caf0c39d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:47.017204 30677 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d00dab869714576ae199f6caf0c39d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d00dab869714576ae199f6caf0c39d8, State: Initialized, Role: FOLLOWER
I20260620 18:38:47.017864 30677 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d00dab869714576ae199f6caf0c39d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8279b148b2454bfb90afa089240c8086" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 36693 } } peers { permanent_uuid: "0d00dab869714576ae199f6caf0c39d8" member_type: VOTER last_known_addr { host: "127.23.115.61" port: 45651 } }
I20260620 18:38:47.018826 30684 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d00dab869714576ae199f6caf0c39d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8279b148b2454bfb90afa089240c8086" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 36693 } } peers { permanent_uuid: "0d00dab869714576ae199f6caf0c39d8" member_type: VOTER last_known_addr { host: "127.23.115.61" port: 45651 } } }
I20260620 18:38:47.019279 30684 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d00dab869714576ae199f6caf0c39d8 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:38:47.019975 30677 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d00dab869714576ae199f6caf0c39d8 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 18:38:47.030464 30695 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0d00dab869714576ae199f6caf0c39d8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 18:38:47.030620 30695 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 18:38:47.033274 30610 master_runner.cc:428] Detected that this master 0d00dab869714576ae199f6caf0c39d8 is joining an existing cluster
I20260620 18:38:47.033380 30610 master_runner.cc:432] Initiating AddMaster RPC to add 127.23.115.61:45651
I20260620 18:38:47.056102 30681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:47.038036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53564 (local address 127.23.115.62:36693)
0620 18:38:47.038235 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:47.038241 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:47.039310 (+ 1069us) server_negotiation.cc:407] Connection header received
0620 18:38:47.039580 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:47.039586 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:47.039669 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:47.039742 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:47.041220 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:47.042330 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:47.044357 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:47.044989 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:47.045091 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:47.047834 (+ 2743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:47.047841 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:47.047846 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:47.047887 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:47.052079 (+ 4192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:47.053169 (+ 1090us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:47.053175 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:47.053179 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:47.053346 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:47.053792 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:47.053797 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:47.053801 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:47.054189 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:47.054318 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:47.055535 (+ 1217us) server_negotiation.cc:299] Negotiation successful
0620 18:38:47.055774 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 18:38:47.057317 30701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:47.038201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:36693 (local address 127.0.0.1:53564)
0620 18:38:47.039204 (+ 1003us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:47.039226 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:47.039425 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:47.039959 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:47.039964 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:47.040622 (+ 658us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:47.041024 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:47.041042 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:47.042520 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:47.042525 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:47.044184 (+ 1659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:47.044199 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:47.045490 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:47.045499 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:47.045686 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:47.046349 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:47.046372 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:47.047632 (+ 1260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:47.052245 (+ 4613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:47.052249 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:47.052253 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:47.053003 (+ 750us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:47.053482 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:47.053491 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:47.053495 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:47.053659 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:47.054376 (+ 717us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:47.054381 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:47.054588 (+ 207us) client_negotiation.cc:769] Sending connection context
0620 18:38:47.054791 (+ 203us) client_negotiation.cc:240] Negotiation successful
0620 18:38:47.055563 (+ 772us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":803,"thread_start_us":606,"threads_started":1}
WARNING: no policy specified for oryx/127.23.115.60@KRBTEST.COM; defaulting to no policy
I20260620 18:38:47.061448 30680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:47.039719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45608 (local address 127.23.115.61:45651)
0620 18:38:47.039865 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:47.039870 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:47.045878 (+ 6008us) server_negotiation.cc:407] Connection header received
0620 18:38:47.046133 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:47.046138 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:47.046207 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:47.046271 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:47.047642 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:47.048662 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:47.051026 (+ 2364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:47.051762 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:47.051878 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:47.054219 (+ 2341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:47.054225 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:47.054230 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:47.054262 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:47.058551 (+ 4289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:47.059322 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:47.059327 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:47.059329 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:47.059467 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:47.059861 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:47.059865 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:47.059868 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:47.060236 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:47.060358 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:47.060852 (+ 494us) server_negotiation.cc:299] Negotiation successful
0620 18:38:47.061046 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 18:38:47.062862 30702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:47.039420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.61:45651 (local address 127.0.0.1:45608)
0620 18:38:47.045777 (+ 6357us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:47.045793 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:47.045985 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:47.046417 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:47.046420 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:47.047139 (+ 719us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:47.047481 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:47.047493 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:47.048877 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:47.048882 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:47.050663 (+ 1781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:47.050698 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:47.052214 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:47.052220 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:47.052403 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:47.053005 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:47.053026 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:47.054041 (+ 1015us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:47.058690 (+ 4649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:47.058695 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:47.058699 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:47.059213 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:47.059587 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:47.059595 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:47.059598 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:47.059752 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:47.060416 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:47.060419 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:47.060554 (+ 135us) client_negotiation.cc:769] Sending connection context
0620 18:38:47.061952 (+ 1398us) client_negotiation.cc:240] Negotiation successful
0620 18:38:47.062368 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6247,"thread_start_us":6053,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.115.60@KRBTEST.COM" created.
Jun 20 18:38:47 dist-test-slave-d645 krb5kdc[30104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980724, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.115.62@KRBTEST.COM for oryx/127.23.115.61@KRBTEST.COM
I20260620 18:38:47.082195 30680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:47.065228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45620 (local address 127.23.115.61:45651)
0620 18:38:47.065371 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:47.065375 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:47.066018 (+ 643us) server_negotiation.cc:407] Connection header received
0620 18:38:47.066278 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:47.066284 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:47.066374 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:47.066469 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:47.067881 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:47.068885 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:47.070751 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:47.071433 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:47.071557 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:47.076203 (+ 4646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:47.076210 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:47.076214 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:47.076255 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:47.079187 (+ 2932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:47.080017 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:47.080021 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:47.080024 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:47.080163 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:47.080525 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:47.080529 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:47.080531 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:47.080884 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:47.081024 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:47.081589 (+ 565us) server_negotiation.cc:299] Negotiation successful
0620 18:38:47.081821 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 18:38:47.082556 30704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:47.065311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.61:45651 (local address 127.0.0.1:45620)
0620 18:38:47.065904 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:47.065933 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:47.066097 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:47.066549 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:47.066558 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:47.067416 (+ 858us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:47.067695 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:47.067709 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:47.069022 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:47.069028 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:47.070600 (+ 1572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:47.070616 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:47.071610 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:47.071616 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:47.071818 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:47.072507 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:47.072536 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:47.075796 (+ 3260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:47.079376 (+ 3580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:47.079395 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:47.079403 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:47.079885 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:47.080274 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:47.080279 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:47.080283 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:47.080416 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:47.081102 (+ 686us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:47.081112 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:47.081297 (+ 185us) client_negotiation.cc:769] Sending connection context
0620 18:38:47.081470 (+ 173us) client_negotiation.cc:240] Negotiation successful
0620 18:38:47.081813 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":260,"threads_started":1}
I20260620 18:38:47.084757 30149 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.23.115.61:45651: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "0d00dab869714576ae199f6caf0c39d8"
member_type: NON_VOTER
last_known_addr {
host: "127.23.115.61"
port: 45651
}
attrs {
promote: true
}
}
dest_uuid: "8279b148b2454bfb90afa089240c8086"
cas_config_opid_index: -1
I20260620 18:38:47.085780 30149 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8279b148b2454bfb90afa089240c8086 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8279b148b2454bfb90afa089240c8086" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 36693 } } peers { permanent_uuid: "0d00dab869714576ae199f6caf0c39d8" member_type: NON_VOTER last_known_addr { host: "127.23.115.61" port: 45651 } attrs { promote: true } }
I20260620 18:38:47.089442 30708 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8279b148b2454bfb90afa089240c8086 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 0d00dab869714576ae199f6caf0c39d8 (127.23.115.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8279b148b2454bfb90afa089240c8086" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 36693 } } peers { permanent_uuid: "0d00dab869714576ae199f6caf0c39d8" member_type: NON_VOTER last_known_addr { host: "127.23.115.61" port: 45651 } attrs { promote: true } } }
I20260620 18:38:47.091528 30652 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d00dab869714576ae199f6caf0c39d8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:47.093834 30708 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.23.115.61:45651
I20260620 18:38:47.094703 30610 master.cc:537] Master@127.23.115.61:45651 shutting down...
I20260620 18:38:47.097494 30652 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0d00dab869714576ae199f6caf0c39d8 [term 1 FOLLOWER]: Refusing update from remote peer 8279b148b2454bfb90afa089240c8086: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.115.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/krb5kdc/oryx_127.23.115.60.keytab.
Entry for principal oryx/127.23.115.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/krb5kdc/oryx_127.23.115.60.keytab.
I20260620 18:38:47.099957 30709 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8279b148b2454bfb90afa089240c8086 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d00dab869714576ae199f6caf0c39d8" member_type: NON_VOTER last_known_addr { host: "127.23.115.61" port: 45651 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:38:47.100241 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.115.60:33051
--webserver_interface=127.23.115.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/krb5kdc/oryx_127.23.115.60.keytab
--principal=oryx/127.23.115.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:34239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.115.62:36693,127.23.115.61:45651,127.23.115.60:33051
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:38:47.102645 30134 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260620 18:38:47.104296 30134 consensus_peers.cc:487] T 00000000000000000000000000000000 P 8279b148b2454bfb90afa089240c8086 -> Peer 0d00dab869714576ae199f6caf0c39d8 (127.23.115.61:45651): Couldn't send request to peer 0d00dab869714576ae199f6caf0c39d8. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 18:38:47.110364 30707 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8279b148b2454bfb90afa089240c8086 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "8279b148b2454bfb90afa089240c8086" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8279b148b2454bfb90afa089240c8086" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 36693 } } peers { permanent_uuid: "0d00dab869714576ae199f6caf0c39d8" member_type: NON_VOTER last_known_addr { host: "127.23.115.61" port: 45651 } attrs { promote: true } } }
I20260620 18:38:47.110785 30707 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8279b148b2454bfb90afa089240c8086 [sys.catalog]: This master's current role is: LEADER
I20260620 18:38:47.111418 30610 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 0d00dab869714576ae199f6caf0c39d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:38:47.111874 30610 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 0d00dab869714576ae199f6caf0c39d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:38:47.112073 30610 tablet_replica.cc:330] T 00000000000000000000000000000000 P 0d00dab869714576ae199f6caf0c39d8: stopping tablet replica
I20260620 18:38:47.118534 30610 master.cc:552] Master@127.23.115.61:45651 shutdown complete.
I20260620 18:38:47.118639 30610 master_runner.cc:305] Clearing existing system tablet
I20260620 18:38:47.119238 30610 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:47.119344 30610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:47.122525 30610 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 18:38:47.124382 30714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:47.124769 30610 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:47.125247 30610 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260620 18:38:47.125391 30610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/master-1/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/master-1/wal
uuid: "0d00dab869714576ae199f6caf0c39d8"
format_stamp: "Formatted at 2026-06-20 18:38:46 on dist-test-slave-d645"
server_key: "1dc7c1c901e9941b4a685355d7cea31e"
server_key_iv: "0f1bc00141961b90ce7ee742b1ea39aa"
server_key_version: "encryptionkey@0"
I20260620 18:38:47.130533 30610 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 0d00dab869714576ae199f6caf0c39d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:38:47.137887 30610 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 0d00dab869714576ae199f6caf0c39d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 18:38:47.138193 30610 log.cc:1196] T 00000000000000000000000000000000 P 0d00dab869714576ae199f6caf0c39d8: Deleting WAL directory at /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260620 18:38:47.138696 30610 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 0d00dab869714576ae199f6caf0c39d8: Deleting consensus metadata
I20260620 18:38:47.139048 30610 master_runner.cc:315] Copying system tablet from 127.23.115.62:36693
I20260620 18:38:47.141656 30610 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 0d00dab869714576ae199f6caf0c39d8: tablet copy: Beginning tablet copy session from remote peer at address 127.23.115.62:36693
I20260620 18:38:47.158453 30681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:47.142526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53574 (local address 127.23.115.62:36693)
0620 18:38:47.142707 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:47.142714 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:47.143111 (+ 397us) server_negotiation.cc:407] Connection header received
0620 18:38:47.143385 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:47.143392 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:47.143514 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:47.143631 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:47.144615 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:47.145802 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:47.147682 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:47.148296 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:47.148430 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:47.151025 (+ 2595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:47.151051 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:47.151059 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:47.151138 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:47.155337 (+ 4199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:47.156222 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:47.156228 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:47.156232 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:47.156425 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:47.156833 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:47.156838 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:47.156841 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:47.157236 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:47.157393 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:47.157822 (+ 429us) server_negotiation.cc:299] Negotiation successful
0620 18:38:47.158067 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 18:38:47.158510 30720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:47.142475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:36693 (local address 127.0.0.1:53574)
0620 18:38:47.143000 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:47.143019 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:47.143228 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:47.143666 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:47.143672 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:47.144192 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:47.144474 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:47.144486 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:47.145960 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:47.145966 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:47.147510 (+ 1544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:47.147525 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:47.148458 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:47.148464 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:47.148646 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:47.149422 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:47.149447 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:47.150552 (+ 1105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:47.155520 (+ 4968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:47.155527 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:47.155531 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:47.156067 (+ 536us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:47.156542 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:47.156547 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:47.156551 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:47.156727 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:47.157361 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:47.157366 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:47.157551 (+ 185us) client_negotiation.cc:769] Sending connection context
0620 18:38:47.157725 (+ 174us) client_negotiation.cc:240] Negotiation successful
0620 18:38:47.158058 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":243,"threads_started":1}
I20260620 18:38:47.160352 30169 tablet_copy_service.cc:140] P 8279b148b2454bfb90afa089240c8086: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 0d00dab869714576ae199f6caf0c39d8 ({username='oryx', principal='oryx/127.23.115.61@KRBTEST.COM'} at 127.0.0.1:53574)
I20260620 18:38:47.160733 30169 tablet_copy_service.cc:161] P 8279b148b2454bfb90afa089240c8086: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 0d00dab869714576ae199f6caf0c39d8 at {username='oryx', principal='oryx/127.23.115.61@KRBTEST.COM'} at 127.0.0.1:53574: session id = 0d00dab869714576ae199f6caf0c39d8-00000000000000000000000000000000
I20260620 18:38:47.167130 30169 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 8279b148b2454bfb90afa089240c8086: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 18:38:47.169550 30610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 18:38:47.177357 30610 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 0d00dab869714576ae199f6caf0c39d8: tablet copy: Starting download of 0 data blocks...
I20260620 18:38:47.177693 30610 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 0d00dab869714576ae199f6caf0c39d8: tablet copy: Starting download of 1 WAL segments...
I20260620 18:38:47.180596 30610 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 0d00dab869714576ae199f6caf0c39d8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 18:38:47.185163 30169 tablet_copy_service.cc:342] P 8279b148b2454bfb90afa089240c8086: Request end of tablet copy session 0d00dab869714576ae199f6caf0c39d8-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.23.115.61@KRBTEST.COM'} at 127.0.0.1:53574
I20260620 18:38:47.185385 30169 tablet_copy_service.cc:433] P 8279b148b2454bfb90afa089240c8086: ending tablet copy session 0d00dab869714576ae199f6caf0c39d8-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 0d00dab869714576ae199f6caf0c39d8
I20260620 18:38:47.195612 30610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:47.198598 30721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:47.203800 30722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:47.211575 30724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:47.212109 30610 server_base.cc:1034] running on GCE node
I20260620 18:38:47.212450 30610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:47.213045 30610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:47.214255 30610 hybrid_clock.cc:648] HybridClock initialized: now 1781980727214196 us; error 35 us; skew 500 ppm
I20260620 18:38:47.214473 30610 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:38:47 dist-test-slave-d645 krb5kdc[30104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980727, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.115.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:47.216187 30610 init.cc:377] Logged in from keytab as oryx/127.23.115.61@KRBTEST.COM (short username oryx)
I20260620 18:38:47.219807 30610 webserver.cc:460] Webserver started at http://127.23.115.61:41231/ using document root <none> and password file <none>
I20260620 18:38:47.220091 30610 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:47.220160 30610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:47.222999 30610 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 18:38:47.224515 30731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:47.224882 30610 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:38:47.225023 30610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/master-1/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/master-1/wal
uuid: "0d00dab869714576ae199f6caf0c39d8"
format_stamp: "Formatted at 2026-06-20 18:38:46 on dist-test-slave-d645"
server_key: "1dc7c1c901e9941b4a685355d7cea31e"
server_key_iv: "0f1bc00141961b90ce7ee742b1ea39aa"
server_key_version: "encryptionkey@0"
I20260620 18:38:47.225172 30610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:47.240597 30610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:47.243762 30610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:47.282588 30610 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.61:45651
I20260620 18:38:47.282671 30782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.61:45651 every 8 connection(s)
I20260620 18:38:47.283987 30610 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/master-1/data/info.pb
I20260620 18:38:47.289481 30783 sys_catalog.cc:263] Verifying existing consensus state
I20260620 18:38:47.293708 30783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d00dab869714576ae199f6caf0c39d8: Bootstrap starting.
I20260620 18:38:47.313602 30783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d00dab869714576ae199f6caf0c39d8: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 18:38:47.314393 30783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d00dab869714576ae199f6caf0c39d8: Bootstrap complete.
I20260620 18:38:47.315696 30783 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d00dab869714576ae199f6caf0c39d8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8279b148b2454bfb90afa089240c8086" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 36693 } } peers { permanent_uuid: "0d00dab869714576ae199f6caf0c39d8" member_type: NON_VOTER last_known_addr { host: "127.23.115.61" port: 45651 } attrs { promote: true } }
I20260620 18:38:47.315999 30783 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d00dab869714576ae199f6caf0c39d8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0d00dab869714576ae199f6caf0c39d8, State: Initialized, Role: LEARNER
I20260620 18:38:47.316394 30783 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d00dab869714576ae199f6caf0c39d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8279b148b2454bfb90afa089240c8086" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 36693 } } peers { permanent_uuid: "0d00dab869714576ae199f6caf0c39d8" member_type: NON_VOTER last_known_addr { host: "127.23.115.61" port: 45651 } attrs { promote: true } }
I20260620 18:38:47.317397 30786 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d00dab869714576ae199f6caf0c39d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8279b148b2454bfb90afa089240c8086" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 36693 } } peers { permanent_uuid: "0d00dab869714576ae199f6caf0c39d8" member_type: NON_VOTER last_known_addr { host: "127.23.115.61" port: 45651 } attrs { promote: true } } }
I20260620 18:38:47.317800 30783 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d00dab869714576ae199f6caf0c39d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:38:47.317840 30786 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d00dab869714576ae199f6caf0c39d8 [sys.catalog]: This master's current role is: LEARNER
I20260620 18:38:47.327872 30797 catalog_manager.cc:1259] Loaded cluster ID: f0357000094b4f94b558a6c421dae89d
I20260620 18:38:47.327972 30797 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0d00dab869714576ae199f6caf0c39d8: loading cluster ID for follower catalog manager: success
I20260620 18:38:47.331314 30797 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0d00dab869714576ae199f6caf0c39d8: acquiring CA information for follower catalog manager: success
I20260620 18:38:47.333559 30797 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0d00dab869714576ae199f6caf0c39d8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 18:38:47.437680 30605 heartbeater.cc:503] Master 127.23.115.62:36693 was elected leader, sending a full tablet report...
W20260620 18:38:47.498265 30711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:47.498554 30711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:47.498682 30711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:47.505942 30711 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:38:47.506102 30711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:47.506183 30711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:47.506259 30711 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:38:47.506317 30711 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:38:47.516294 30711 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:34239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.23.115.62:36693,127.23.115.61:45651,127.23.115.60:33051
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/krb5kdc/oryx_127.23.115.60.keytab
--principal=oryx/127.23.115.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.115.60:33051
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.23.115.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:38:47.517624 30711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:47.518972 30711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:47.531208 30803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:47.531167 30805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:47.531252 30802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:47.532305 30711 server_base.cc:1034] running on GCE node
I20260620 18:38:47.533079 30711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:47.534479 30711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:47.535677 30711 hybrid_clock.cc:648] HybridClock initialized: now 1781980727535597 us; error 81 us; skew 500 ppm
I20260620 18:38:47.535995 30711 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:38:47 dist-test-slave-d645 krb5kdc[30104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980727, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.115.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:47.538980 30711 init.cc:377] Logged in from keytab as oryx/127.23.115.60@KRBTEST.COM (short username oryx)
I20260620 18:38:47.540513 30711 webserver.cc:460] Webserver started at http://127.23.115.60:37553/ using document root <none> and password file <none>
I20260620 18:38:47.541267 30711 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:47.541384 30711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:47.541656 30711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:38:47.544577 30711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/master-2/data/instance:
uuid: "294609dc6da34eefbb57726f025c5619"
format_stamp: "Formatted at 2026-06-20 18:38:47 on dist-test-slave-d645"
server_key: "63333fad221566b28c4a55f7a79c1317"
server_key_iv: "7f1f26e0fd9f5711ae5db60d211d48ed"
server_key_version: "encryptionkey@0"
I20260620 18:38:47.545347 30711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/master-2/wal/instance:
uuid: "294609dc6da34eefbb57726f025c5619"
format_stamp: "Formatted at 2026-06-20 18:38:47 on dist-test-slave-d645"
server_key: "63333fad221566b28c4a55f7a79c1317"
server_key_iv: "7f1f26e0fd9f5711ae5db60d211d48ed"
server_key_version: "encryptionkey@0"
I20260620 18:38:47.550578 30711 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 18:38:47.554093 30812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:47.555512 30711 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 18:38:47.555671 30711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/master-2/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/master-2/wal
uuid: "294609dc6da34eefbb57726f025c5619"
format_stamp: "Formatted at 2026-06-20 18:38:47 on dist-test-slave-d645"
server_key: "63333fad221566b28c4a55f7a79c1317"
server_key_iv: "7f1f26e0fd9f5711ae5db60d211d48ed"
server_key_version: "encryptionkey@0"
I20260620 18:38:47.555851 30711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:47.584583 30711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:47.589028 30711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:47.589332 30711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:47.608800 30711 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.60:33051
I20260620 18:38:47.608850 30863 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.60:33051 every 8 connection(s)
I20260620 18:38:47.610205 30711 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/master-2/data/info.pb
I20260620 18:38:47.614043 30864 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:47.620333 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 30711
I20260620 18:38:47.621152 30864 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.115.62" port: 36693 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:38:47 dist-test-slave-d645 krb5kdc[30104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980727, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.115.60@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM
Jun 20 18:38:47 dist-test-slave-d645 krb5kdc[30104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980724, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.115.60@KRBTEST.COM
I20260620 18:38:47.641568 30681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:47.624284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46990 (local address 127.23.115.62:36693)
0620 18:38:47.624474 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:47.624480 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:47.625394 (+ 914us) server_negotiation.cc:407] Connection header received
0620 18:38:47.625890 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:47.625895 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:47.625970 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:47.626100 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:47.628000 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:47.629097 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:47.631251 (+ 2154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:47.631771 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:47.631901 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:47.635389 (+ 3488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:47.635394 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:47.635398 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:47.635430 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:47.638416 (+ 2986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:47.639308 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:47.639314 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:47.639318 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:47.639468 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:47.639888 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:47.639891 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:47.639894 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:47.640176 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:47.640311 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:47.641012 (+ 701us) server_negotiation.cc:299] Negotiation successful
0620 18:38:47.641233 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 18:38:47.642599 30867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:47.622892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38284 (local address 127.23.115.60:33051)
0620 18:38:47.623646 (+ 754us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:47.623657 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:47.623693 (+ 36us) server_negotiation.cc:407] Connection header received
0620 18:38:47.625660 (+ 1967us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:47.625681 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:47.625933 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:47.626125 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:47.628148 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:47.629642 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:47.631456 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:47.632296 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:47.632406 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:47.636531 (+ 4125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:47.636543 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:47.636558 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:47.636594 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:47.639696 (+ 3102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:47.640380 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:47.640386 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:47.640389 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:47.640514 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:47.640858 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:47.640862 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:47.640864 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:47.641188 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:47.641478 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:47.642039 (+ 561us) server_negotiation.cc:299] Negotiation successful
0620 18:38:47.642233 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":571,"spinlock_wait_cycles":26368,"thread_start_us":314,"threads_started":1}
I20260620 18:38:47.642639 30868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:47.624286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:36693 (local address 127.0.0.1:46990)
0620 18:38:47.625212 (+ 926us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:47.625289 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:47.625662 (+ 373us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:47.626205 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:47.626213 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:47.627268 (+ 1055us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:47.627773 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:47.627798 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:47.629274 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:47.629278 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:47.631071 (+ 1793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:47.631086 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:47.632007 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:47.632015 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:47.632352 (+ 337us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:47.633011 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:47.633043 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:47.635200 (+ 2157us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:47.638555 (+ 3355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:47.638566 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:47.638575 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:47.639174 (+ 599us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:47.639569 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:47.639572 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:47.639574 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:47.639763 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:47.640301 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:47.640309 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:47.640645 (+ 336us) client_negotiation.cc:769] Sending connection context
0620 18:38:47.640893 (+ 248us) client_negotiation.cc:240] Negotiation successful
0620 18:38:47.641406 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":669,"mutex_wait_us":1,"thread_start_us":492,"threads_started":1}
I20260620 18:38:47.644096 30864 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.115.61" port: 45651 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:38:47 dist-test-slave-d645 krb5kdc[30104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980727, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.115.60@KRBTEST.COM for oryx/127.23.115.61@KRBTEST.COM
I20260620 18:38:47.657413 30868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:47.644704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.61:45651 (local address 127.0.0.1:34552)
0620 18:38:47.644888 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:47.644902 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:47.644975 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:47.645656 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:47.645659 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:47.645965 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:47.646092 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:47.646099 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:47.647547 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:47.647550 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:47.648482 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:47.648493 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:47.649199 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:47.649203 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:47.649354 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:47.649936 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:47.649955 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:47.651991 (+ 2036us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:47.655062 (+ 3071us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:47.655077 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:47.655081 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:47.655496 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:47.655952 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:47.655957 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:47.655960 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:47.656109 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:47.656688 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:47.656691 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:47.656815 (+ 124us) client_negotiation.cc:769] Sending connection context
0620 18:38:47.656936 (+ 121us) client_negotiation.cc:240] Negotiation successful
0620 18:38:47.657139 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260620 18:38:47.657579 30869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:47.644890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34552 (local address 127.23.115.61:45651)
0620 18:38:47.645397 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:47.645403 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:47.645421 (+ 18us) server_negotiation.cc:407] Connection header received
0620 18:38:47.645468 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:47.645471 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:47.645525 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:47.645603 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:47.646279 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:47.647420 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:47.648625 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:47.649059 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:47.649159 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:47.652195 (+ 3036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:47.652209 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:47.652212 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:47.652245 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:47.654672 (+ 2427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:47.655591 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:47.655595 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:47.655597 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:47.655737 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:47.656235 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:47.656241 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:47.656244 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:47.656530 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:47.656652 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:47.657072 (+ 420us) server_negotiation.cc:299] Negotiation successful
0620 18:38:47.657247 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":419,"threads_started":1}
I20260620 18:38:47.658685 30864 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.115.60" port: 33051 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:38:47 dist-test-slave-d645 krb5kdc[30104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980727, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.115.60@KRBTEST.COM for oryx/127.23.115.60@KRBTEST.COM
I20260620 18:38:47.673115 30868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:47.659428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.60:33051 (local address 127.0.0.1:38294)
0620 18:38:47.659690 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:47.659708 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:47.659875 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:47.660224 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:47.660229 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:47.660659 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:47.660847 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:47.660860 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:47.661827 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:47.661833 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:47.662786 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:47.662799 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:47.663587 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:47.663593 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:47.663838 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:47.664394 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:47.664412 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:47.666631 (+ 2219us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:47.670119 (+ 3488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:47.670123 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:47.670126 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:47.670442 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:47.671062 (+ 620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:47.671066 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:47.671069 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:47.671258 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:47.671976 (+ 718us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:47.671983 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:47.672191 (+ 208us) client_negotiation.cc:769] Sending connection context
0620 18:38:47.672407 (+ 216us) client_negotiation.cc:240] Negotiation successful
0620 18:38:47.672703 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":3}
I20260620 18:38:47.673174 30867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:47.659626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38294 (local address 127.23.115.60:33051)
0620 18:38:47.659807 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:47.659812 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:47.659828 (+ 16us) server_negotiation.cc:407] Connection header received
0620 18:38:47.660043 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:47.660049 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:47.660107 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:47.660192 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:47.660985 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:47.661702 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:47.662924 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:47.663450 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:47.663576 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:47.666811 (+ 3235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:47.666817 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:47.666820 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:47.666847 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:47.669967 (+ 3120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:47.670555 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:47.670560 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:47.670562 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:47.670689 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:47.671373 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:47.671377 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:47.671379 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:47.671647 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:47.671845 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:47.672538 (+ 693us) server_negotiation.cc:299] Negotiation successful
0620 18:38:47.672851 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":108,"server-negotiator.queue_time_us":67}
I20260620 18:38:47.677984 30864 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 294609dc6da34eefbb57726f025c5619: Bootstrap starting.
I20260620 18:38:47.681034 30864 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 294609dc6da34eefbb57726f025c5619: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:47.682367 30864 log.cc:824] T 00000000000000000000000000000000 P 294609dc6da34eefbb57726f025c5619: Log is configured to *not* fsync() on all Append() calls
I20260620 18:38:47.685257 30864 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 294609dc6da34eefbb57726f025c5619: No bootstrap required, opened a new log
I20260620 18:38:47.688560 30869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:47.683203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34562 (local address 127.23.115.61:45651)
0620 18:38:47.683374 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:47.683379 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:47.683684 (+ 305us) server_negotiation.cc:407] Connection header received
0620 18:38:47.683863 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:47.683866 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:47.683958 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:47.684050 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:38:47.684425 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:47.685109 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:47.686781 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:47.687650 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:47.687775 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:47.687968 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:47.688184 (+ 216us) server_negotiation.cc:299] Negotiation successful
0620 18:38:47.688319 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 18:38:47.688596 30872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:47.683111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.61:45651 (local address 127.0.0.1:34562)
0620 18:38:47.683584 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:47.683599 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:47.683729 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:47.684099 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:47.684104 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:47.684139 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:38:47.684295 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:47.684303 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:47.685240 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:47.685246 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:47.686638 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:47.686651 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:47.687815 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:47.687821 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:47.688013 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:47.688065 (+ 52us) client_negotiation.cc:769] Sending connection context
0620 18:38:47.688158 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 18:38:47.688297 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":259,"threads_started":1}
I20260620 18:38:47.689512 30758 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 0d00dab869714576ae199f6caf0c39d8 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260620 18:38:47.689993 30786 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d00dab869714576ae199f6caf0c39d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8279b148b2454bfb90afa089240c8086. Latest consensus state: current_term: 1 leader_uuid: "8279b148b2454bfb90afa089240c8086" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8279b148b2454bfb90afa089240c8086" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 36693 } } peers { permanent_uuid: "0d00dab869714576ae199f6caf0c39d8" member_type: NON_VOTER last_known_addr { host: "127.23.115.61" port: 45651 } attrs { promote: true } } }
I20260620 18:38:47.690253 30786 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d00dab869714576ae199f6caf0c39d8 [sys.catalog]: This master's current role is: LEARNER
I20260620 18:38:47.692953 30864 raft_consensus.cc:348] T 00000000000000000000000000000000 P 294609dc6da34eefbb57726f025c5619 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8279b148b2454bfb90afa089240c8086" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 36693 } } peers { permanent_uuid: "0d00dab869714576ae199f6caf0c39d8" member_type: VOTER last_known_addr { host: "127.23.115.61" port: 45651 } } peers { permanent_uuid: "294609dc6da34eefbb57726f025c5619" member_type: VOTER last_known_addr { host: "127.23.115.60" port: 33051 } }
I20260620 18:38:47.693259 30864 raft_consensus.cc:374] T 00000000000000000000000000000000 P 294609dc6da34eefbb57726f025c5619 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:47.693367 30864 raft_consensus.cc:729] T 00000000000000000000000000000000 P 294609dc6da34eefbb57726f025c5619 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 294609dc6da34eefbb57726f025c5619, State: Initialized, Role: FOLLOWER
I20260620 18:38:47.693919 30864 consensus_queue.cc:260] T 00000000000000000000000000000000 P 294609dc6da34eefbb57726f025c5619 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8279b148b2454bfb90afa089240c8086" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 36693 } } peers { permanent_uuid: "0d00dab869714576ae199f6caf0c39d8" member_type: VOTER last_known_addr { host: "127.23.115.61" port: 45651 } } peers { permanent_uuid: "294609dc6da34eefbb57726f025c5619" member_type: VOTER last_known_addr { host: "127.23.115.60" port: 33051 } }
I20260620 18:38:47.695178 30873 sys_catalog.cc:455] T 00000000000000000000000000000000 P 294609dc6da34eefbb57726f025c5619 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8279b148b2454bfb90afa089240c8086" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 36693 } } peers { permanent_uuid: "0d00dab869714576ae199f6caf0c39d8" member_type: VOTER last_known_addr { host: "127.23.115.61" port: 45651 } } peers { permanent_uuid: "294609dc6da34eefbb57726f025c5619" member_type: VOTER last_known_addr { host: "127.23.115.60" port: 33051 } } }
I20260620 18:38:47.695647 30873 sys_catalog.cc:458] T 00000000000000000000000000000000 P 294609dc6da34eefbb57726f025c5619 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:38:47.696734 30864 sys_catalog.cc:564] T 00000000000000000000000000000000 P 294609dc6da34eefbb57726f025c5619 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 18:38:47.707125 30884 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 294609dc6da34eefbb57726f025c5619: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 18:38:47.707312 30884 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 18:38:47.712141 30711 master_runner.cc:428] Detected that this master 294609dc6da34eefbb57726f025c5619 is joining an existing cluster
I20260620 18:38:47.712294 30711 master_runner.cc:432] Initiating AddMaster RPC to add 127.23.115.60:33051
I20260620 18:38:47.733183 30681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:47.716517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46992 (local address 127.23.115.62:36693)
0620 18:38:47.716739 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:47.716745 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:47.717307 (+ 562us) server_negotiation.cc:407] Connection header received
0620 18:38:47.717587 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:47.717592 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:47.717678 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:47.717752 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:47.719247 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:47.720335 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:47.723311 (+ 2976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:47.723936 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:47.724021 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:47.726608 (+ 2587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:47.726615 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:47.726619 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:47.726659 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:47.729950 (+ 3291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:47.730754 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:47.730760 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:47.730763 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:47.730916 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:47.731432 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:47.731444 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:47.731447 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:47.731818 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:47.731935 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:47.732596 (+ 661us) server_negotiation.cc:299] Negotiation successful
0620 18:38:47.732840 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 18:38:47.734262 30890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:47.716536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:36693 (local address 127.0.0.1:46992)
0620 18:38:47.717190 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:47.717209 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:47.717427 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:47.717968 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:47.717973 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:47.718676 (+ 703us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:47.719066 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:47.719079 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:47.720481 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:47.720485 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:47.723150 (+ 2665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:47.723164 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:47.724301 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:47.724307 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:47.724486 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:47.725174 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:47.725196 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:47.726389 (+ 1193us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:47.730106 (+ 3717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:47.730111 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:47.730115 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:47.730620 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:47.731093 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:47.731098 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:47.731101 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:47.731298 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:47.732029 (+ 731us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:47.732034 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:47.732309 (+ 275us) client_negotiation.cc:769] Sending connection context
0620 18:38:47.733458 (+ 1149us) client_negotiation.cc:240] Negotiation successful
0620 18:38:47.733779 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"mutex_wait_us":34,"thread_start_us":340,"threads_started":1}
I20260620 18:38:47.734375 30869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:47.717543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34570 (local address 127.23.115.61:45651)
0620 18:38:47.717759 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:47.717770 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:47.718366 (+ 596us) server_negotiation.cc:407] Connection header received
0620 18:38:47.718611 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:47.718616 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:47.718705 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:47.718778 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:47.720623 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:47.721708 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:47.724061 (+ 2353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:47.724678 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:47.724782 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:47.727405 (+ 2623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:47.727413 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:47.727418 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:47.727459 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:47.731400 (+ 3941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:47.732244 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:47.732252 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:47.732256 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:47.732400 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:47.732839 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:47.732843 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:47.732846 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:47.733147 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:47.733268 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:47.733844 (+ 576us) server_negotiation.cc:299] Negotiation successful
0620 18:38:47.734072 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260620 18:38:47.735677 30891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:47.717302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.61:45651 (local address 127.0.0.1:34570)
0620 18:38:47.718276 (+ 974us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:47.718293 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:47.718458 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:47.719083 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:47.719088 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:47.719987 (+ 899us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:47.720454 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:47.720466 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:47.722209 (+ 1743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:47.722213 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:47.723900 (+ 1687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:47.723914 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:47.725059 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:47.725066 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:47.725289 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:47.725934 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:47.725960 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:47.727191 (+ 1231us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:47.731558 (+ 4367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:47.731562 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:47.731565 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:47.732115 (+ 550us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:47.732527 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:47.732531 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:47.732534 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:47.732701 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:47.733345 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:47.733349 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:47.733561 (+ 212us) client_negotiation.cc:769] Sending connection context
0620 18:38:47.734592 (+ 1031us) client_negotiation.cc:240] Negotiation successful
0620 18:38:47.734970 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":843,"mutex_wait_us":76,"thread_start_us":670,"threads_started":1}
I20260620 18:38:47.736850 30892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:47.718320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.60:33051 (local address 127.0.0.1:38310)
0620 18:38:47.719633 (+ 1313us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:47.719649 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:47.719767 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:47.720068 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:47.720071 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:47.720789 (+ 718us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:47.721152 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:47.721165 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:47.722762 (+ 1597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:47.722768 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:47.724347 (+ 1579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:47.724365 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:47.725766 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:47.725773 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:47.725961 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:47.726542 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:47.726587 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:47.727889 (+ 1302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:47.732283 (+ 4394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:47.732287 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:47.732294 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:47.732781 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:47.733182 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:47.733187 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:47.733190 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:47.733661 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:47.734826 (+ 1165us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:47.734833 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:47.735049 (+ 216us) client_negotiation.cc:769] Sending connection context
0620 18:38:47.736031 (+ 982us) client_negotiation.cc:240] Negotiation successful
0620 18:38:47.736342 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1218,"spinlock_wait_cycles":38272,"thread_start_us":1057,"threads_started":1}
I20260620 18:38:47.736881 30867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:47.718731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38310 (local address 127.23.115.60:33051)
0620 18:38:47.718894 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:47.718900 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:47.719721 (+ 821us) server_negotiation.cc:407] Connection header received
0620 18:38:47.719852 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:47.719857 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:47.719921 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:47.719994 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:47.721381 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:47.722578 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:47.724504 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:47.725271 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:47.725385 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:47.728086 (+ 2701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:47.728093 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:47.728098 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:47.728130 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:47.732137 (+ 4007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:47.732896 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:47.732901 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:47.732904 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:47.733057 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:47.733783 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:47.733789 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:47.733792 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:47.734227 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:47.734746 (+ 519us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:47.736155 (+ 1409us) server_negotiation.cc:299] Negotiation successful
0620 18:38:47.736502 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":165,"server-negotiator.queue_time_us":66}
Jun 20 18:38:47 dist-test-slave-d645 krb5kdc[30104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980724, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.115.62@KRBTEST.COM for oryx/127.23.115.60@KRBTEST.COM
I20260620 18:38:47.749399 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 30210
I20260620 18:38:47.755213 30872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:47.739835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.60:33051 (local address 127.0.0.1:38324)
0620 18:38:47.739998 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:47.740016 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:47.740124 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:47.740493 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:47.740498 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:47.740893 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:47.741033 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:47.741041 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:47.742060 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:47.742066 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:47.743054 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:47.743066 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:47.743781 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:47.743787 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:47.743957 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:47.744480 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:47.744503 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:47.746740 (+ 2237us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:47.751890 (+ 5150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:47.751897 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:47.751903 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:47.752567 (+ 664us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:47.753017 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:47.753023 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:47.753027 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:47.753186 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:47.753864 (+ 678us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:47.753869 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:47.754082 (+ 213us) client_negotiation.cc:769] Sending connection context
0620 18:38:47.754214 (+ 132us) client_negotiation.cc:240] Negotiation successful
0620 18:38:47.754727 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260620 18:38:47.755273 30867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:47.739852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38324 (local address 127.23.115.60:33051)
0620 18:38:47.739997 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:47.740002 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:47.740080 (+ 78us) server_negotiation.cc:407] Connection header received
0620 18:38:47.740273 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:47.740279 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:47.740372 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:47.740449 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:47.741174 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:47.741942 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:47.743182 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:47.743641 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:47.743752 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:47.746970 (+ 3218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:47.746980 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:47.746985 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:47.747024 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:47.751653 (+ 4629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:47.752725 (+ 1072us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:47.752733 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:47.752737 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:47.752895 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:47.753296 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:47.753302 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:47.753305 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:47.753712 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:47.753893 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:47.754484 (+ 591us) server_negotiation.cc:299] Negotiation successful
0620 18:38:47.754775 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 18:38:47.757418 30149 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.23.115.60:33051: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "294609dc6da34eefbb57726f025c5619"
member_type: NON_VOTER
last_known_addr {
host: "127.23.115.60"
port: 33051
}
attrs {
promote: true
}
}
dest_uuid: "8279b148b2454bfb90afa089240c8086"
cas_config_opid_index: 5
I20260620 18:38:47.758386 30149 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8279b148b2454bfb90afa089240c8086 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8279b148b2454bfb90afa089240c8086" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 36693 } } peers { permanent_uuid: "0d00dab869714576ae199f6caf0c39d8" member_type: NON_VOTER last_known_addr { host: "127.23.115.61" port: 45651 } attrs { promote: true } } peers { permanent_uuid: "294609dc6da34eefbb57726f025c5619" member_type: NON_VOTER last_known_addr { host: "127.23.115.60" port: 33051 } attrs { promote: true } }
I20260620 18:38:47.760006 30870 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8279b148b2454bfb90afa089240c8086 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 294609dc6da34eefbb57726f025c5619 (127.23.115.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8279b148b2454bfb90afa089240c8086" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 36693 } } peers { permanent_uuid: "0d00dab869714576ae199f6caf0c39d8" member_type: NON_VOTER last_known_addr { host: "127.23.115.61" port: 45651 } attrs { promote: true } } peers { permanent_uuid: "294609dc6da34eefbb57726f025c5619" member_type: NON_VOTER last_known_addr { host: "127.23.115.60" port: 33051 } attrs { promote: true } } }
I20260620 18:38:47.761413 30839 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 294609dc6da34eefbb57726f025c5619 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:47.763548 30870 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.23.115.60:33051
I20260620 18:38:47.765259 30839 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 294609dc6da34eefbb57726f025c5619 [term 1 FOLLOWER]: Refusing update from remote peer 8279b148b2454bfb90afa089240c8086: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 18:38:47.767197 30711 master.cc:537] Master@127.23.115.60:33051 shutting down...
Jun 20 18:38:47 dist-test-slave-d645 krb5kdc[30104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980724, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.115.1@KRBTEST.COM
I20260620 18:38:47.789413 30894 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8279b148b2454bfb90afa089240c8086 [LEADER]: Connected to new peer: Peer: permanent_uuid: "294609dc6da34eefbb57726f025c5619" member_type: NON_VOTER last_known_addr { host: "127.23.115.60" port: 33051 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:38:47.790844 30758 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0d00dab869714576ae199f6caf0c39d8 [term 1 LEARNER]: Refusing update from remote peer 8279b148b2454bfb90afa089240c8086: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 18:38:47.793792 30894 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8279b148b2454bfb90afa089240c8086 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d00dab869714576ae199f6caf0c39d8" member_type: NON_VOTER last_known_addr { host: "127.23.115.61" port: 45651 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 18:38:47.794564 30893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:47.751211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43606 (local address 127.23.115.1:39357)
0620 18:38:47.762064 (+ 10853us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:47.762075 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:47.762111 (+ 36us) server_negotiation.cc:407] Connection header received
0620 18:38:47.762187 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:47.762196 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:47.762509 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:47.765159 (+ 2650us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:47.765429 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:47.767015 (+ 1586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:47.779356 (+ 12341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:47.780337 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:47.786139 (+ 5802us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:47.786446 (+ 307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:47.786459 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:47.786476 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:47.786515 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:47.790312 (+ 3797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:47.791244 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:47.791254 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:47.791258 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:47.791424 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:47.791876 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:47.791880 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:47.791884 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:47.792325 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:47.793710 (+ 1385us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:47.793878 (+ 168us) server_negotiation.cc:299] Negotiation successful
0620 18:38:47.794161 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10661,"thread_start_us":11463,"threads_started":1}
I20260620 18:38:47.795980 30333 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:38:47.806607 30896 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8279b148b2454bfb90afa089240c8086 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "8279b148b2454bfb90afa089240c8086" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8279b148b2454bfb90afa089240c8086" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 36693 } } peers { permanent_uuid: "0d00dab869714576ae199f6caf0c39d8" member_type: NON_VOTER last_known_addr { host: "127.23.115.61" port: 45651 } attrs { promote: true } } peers { permanent_uuid: "294609dc6da34eefbb57726f025c5619" member_type: NON_VOTER last_known_addr { host: "127.23.115.60" port: 33051 } attrs { promote: true } } }
I20260620 18:38:47.807108 30896 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8279b148b2454bfb90afa089240c8086 [sys.catalog]: This master's current role is: LEADER
I20260620 18:38:47.809300 30758 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0d00dab869714576ae199f6caf0c39d8 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 294609dc6da34eefbb57726f025c5619 (127.23.115.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8279b148b2454bfb90afa089240c8086" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 36693 } } peers { permanent_uuid: "0d00dab869714576ae199f6caf0c39d8" member_type: NON_VOTER last_known_addr { host: "127.23.115.61" port: 45651 } attrs { promote: true } } peers { permanent_uuid: "294609dc6da34eefbb57726f025c5619" member_type: NON_VOTER last_known_addr { host: "127.23.115.60" port: 33051 } attrs { promote: true } } }
I20260620 18:38:47.818078 30711 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 294609dc6da34eefbb57726f025c5619 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:38:47.818636 30711 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 294609dc6da34eefbb57726f025c5619 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:38:47.818859 30711 tablet_replica.cc:330] T 00000000000000000000000000000000 P 294609dc6da34eefbb57726f025c5619: stopping tablet replica
I20260620 18:38:47.824067 30786 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d00dab869714576ae199f6caf0c39d8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8279b148b2454bfb90afa089240c8086" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8279b148b2454bfb90afa089240c8086" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 36693 } } peers { permanent_uuid: "0d00dab869714576ae199f6caf0c39d8" member_type: NON_VOTER last_known_addr { host: "127.23.115.61" port: 45651 } attrs { promote: true } } peers { permanent_uuid: "294609dc6da34eefbb57726f025c5619" member_type: NON_VOTER last_known_addr { host: "127.23.115.60" port: 33051 } attrs { promote: true } } }
I20260620 18:38:47.824422 30786 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d00dab869714576ae199f6caf0c39d8 [sys.catalog]: This master's current role is: LEARNER
I20260620 18:38:47.828063 30711 master.cc:552] Master@127.23.115.60:33051 shutdown complete.
I20260620 18:38:47.828159 30711 master_runner.cc:305] Clearing existing system tablet
I20260620 18:38:47.828526 30711 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:47.828634 30711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:47.830446 30870 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 8279b148b2454bfb90afa089240c8086: attempting to promote NON_VOTER 0d00dab869714576ae199f6caf0c39d8 to VOTER
I20260620 18:38:47.831923 30870 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8279b148b2454bfb90afa089240c8086 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8279b148b2454bfb90afa089240c8086" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 36693 } } peers { permanent_uuid: "0d00dab869714576ae199f6caf0c39d8" member_type: VOTER last_known_addr { host: "127.23.115.61" port: 45651 } attrs { promote: false } } peers { permanent_uuid: "294609dc6da34eefbb57726f025c5619" member_type: NON_VOTER last_known_addr { host: "127.23.115.60" port: 33051 } attrs { promote: true } }
I20260620 18:38:47.835090 30872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:47.834521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.60:33051 (local address 127.0.0.1:38330)
0620 18:38:47.834714 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:47.834836 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.60:33051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 18:38:47.835930 30758 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0d00dab869714576ae199f6caf0c39d8 [term 1 LEARNER]: Refusing update from remote peer 8279b148b2454bfb90afa089240c8086: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260620 18:38:47.836689 30895 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8279b148b2454bfb90afa089240c8086 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d00dab869714576ae199f6caf0c39d8" member_type: VOTER last_known_addr { host: "127.23.115.61" port: 45651 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260620 18:38:47.839498 30896 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8279b148b2454bfb90afa089240c8086 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 0d00dab869714576ae199f6caf0c39d8 (127.23.115.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8279b148b2454bfb90afa089240c8086" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 36693 } } peers { permanent_uuid: "0d00dab869714576ae199f6caf0c39d8" member_type: VOTER last_known_addr { host: "127.23.115.61" port: 45651 } attrs { promote: false } } peers { permanent_uuid: "294609dc6da34eefbb57726f025c5619" member_type: NON_VOTER last_known_addr { host: "127.23.115.60" port: 33051 } attrs { promote: true } } }
I20260620 18:38:47.841050 30711 fs_manager.cc:709] Time spent opening directory manager: real 0.011s user 0.001s sys 0.005s
I20260620 18:38:47.843761 30870 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8279b148b2454bfb90afa089240c8086 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "8279b148b2454bfb90afa089240c8086" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8279b148b2454bfb90afa089240c8086" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 36693 } } peers { permanent_uuid: "0d00dab869714576ae199f6caf0c39d8" member_type: VOTER last_known_addr { host: "127.23.115.61" port: 45651 } attrs { promote: false } } peers { permanent_uuid: "294609dc6da34eefbb57726f025c5619" member_type: NON_VOTER last_known_addr { host: "127.23.115.60" port: 33051 } attrs { promote: true } } }
I20260620 18:38:47.844010 30870 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8279b148b2454bfb90afa089240c8086 [sys.catalog]: This master's current role is: LEADER
I20260620 18:38:47.844588 30895 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8279b148b2454bfb90afa089240c8086 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8279b148b2454bfb90afa089240c8086" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8279b148b2454bfb90afa089240c8086" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 36693 } } peers { permanent_uuid: "0d00dab869714576ae199f6caf0c39d8" member_type: VOTER last_known_addr { host: "127.23.115.61" port: 45651 } attrs { promote: false } } peers { permanent_uuid: "294609dc6da34eefbb57726f025c5619" member_type: NON_VOTER last_known_addr { host: "127.23.115.60" port: 33051 } attrs { promote: true } } }
I20260620 18:38:47.844903 30895 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8279b148b2454bfb90afa089240c8086 [sys.catalog]: This master's current role is: LEADER
I20260620 18:38:47.845772 30758 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0d00dab869714576ae199f6caf0c39d8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 0d00dab869714576ae199f6caf0c39d8 (127.23.115.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8279b148b2454bfb90afa089240c8086" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 36693 } } peers { permanent_uuid: "0d00dab869714576ae199f6caf0c39d8" member_type: VOTER last_known_addr { host: "127.23.115.61" port: 45651 } attrs { promote: false } } peers { permanent_uuid: "294609dc6da34eefbb57726f025c5619" member_type: NON_VOTER last_known_addr { host: "127.23.115.60" port: 33051 } attrs { promote: true } } }
I20260620 18:38:47.850308 30872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:47.849911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.60:33051 (local address 127.0.0.1:38332)
0620 18:38:47.850064 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:47.850151 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.60:33051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260620 18:38:47.850653 30134 consensus_peers.cc:487] T 00000000000000000000000000000000 P 8279b148b2454bfb90afa089240c8086 -> Peer 294609dc6da34eefbb57726f025c5619 (127.23.115.60:33051): Couldn't send request to peer 294609dc6da34eefbb57726f025c5619. Status: Network error: Client connection negotiation failed: client connection to 127.23.115.60:33051: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 18:38:47.851428 30786 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d00dab869714576ae199f6caf0c39d8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8279b148b2454bfb90afa089240c8086" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8279b148b2454bfb90afa089240c8086" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 36693 } } peers { permanent_uuid: "0d00dab869714576ae199f6caf0c39d8" member_type: VOTER last_known_addr { host: "127.23.115.61" port: 45651 } attrs { promote: false } } peers { permanent_uuid: "294609dc6da34eefbb57726f025c5619" member_type: NON_VOTER last_known_addr { host: "127.23.115.60" port: 33051 } attrs { promote: true } } }
I20260620 18:38:47.851691 30786 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d00dab869714576ae199f6caf0c39d8 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:38:47.851904 30902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:47.852293 30711 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:47.852566 30711 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.002s sys 0.008s
I20260620 18:38:47.852676 30711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/master-2/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/master-2/wal
uuid: "294609dc6da34eefbb57726f025c5619"
format_stamp: "Formatted at 2026-06-20 18:38:47 on dist-test-slave-d645"
server_key: "63333fad221566b28c4a55f7a79c1317"
server_key_iv: "7f1f26e0fd9f5711ae5db60d211d48ed"
server_key_version: "encryptionkey@0"
I20260620 18:38:47.858518 30711 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 294609dc6da34eefbb57726f025c5619: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:38:47.865615 30711 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 294609dc6da34eefbb57726f025c5619: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 18:38:47.865798 30711 log.cc:1196] T 00000000000000000000000000000000 P 294609dc6da34eefbb57726f025c5619: Deleting WAL directory at /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260620 18:38:47.866700 30711 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 294609dc6da34eefbb57726f025c5619: Deleting consensus metadata
I20260620 18:38:47.867008 30711 master_runner.cc:315] Copying system tablet from 127.23.115.62:36693
I20260620 18:38:47.872720 30711 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 294609dc6da34eefbb57726f025c5619: tablet copy: Beginning tablet copy session from remote peer at address 127.23.115.62:36693
I20260620 18:38:47.894137 30681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:47.873579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46994 (local address 127.23.115.62:36693)
0620 18:38:47.873795 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:47.873802 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:47.879597 (+ 5795us) server_negotiation.cc:407] Connection header received
0620 18:38:47.879781 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:47.879787 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:47.879888 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:47.879998 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:47.881161 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:47.882289 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:47.884454 (+ 2165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:47.885105 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:47.885225 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:47.887602 (+ 2377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:47.887610 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:47.887615 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:47.887655 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:47.890895 (+ 3240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:47.891677 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:47.891684 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:47.891688 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:47.891834 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:47.892241 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:47.892245 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:47.892248 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:47.892590 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:47.892729 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:47.893591 (+ 862us) server_negotiation.cc:299] Negotiation successful
0620 18:38:47.893814 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260620 18:38:47.894322 30908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:47.873523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:36693 (local address 127.0.0.1:46994)
0620 18:38:47.879430 (+ 5907us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:47.879454 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:47.879642 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:47.880031 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:47.880036 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:47.880580 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:47.880983 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:47.881000 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:47.882435 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:47.882441 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:47.884288 (+ 1847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:47.884304 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:47.885273 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:47.885279 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:47.885463 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:47.886224 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:47.886248 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:47.887383 (+ 1135us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:47.891053 (+ 3670us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:47.891059 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:47.891063 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:47.891544 (+ 481us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:47.891940 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:47.891945 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:47.891949 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:47.892128 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:47.892802 (+ 674us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:47.892810 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:47.893122 (+ 312us) client_negotiation.cc:769] Sending connection context
0620 18:38:47.893474 (+ 352us) client_negotiation.cc:240] Negotiation successful
0620 18:38:47.893860 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5756,"thread_start_us":5604,"threads_started":1}
I20260620 18:38:47.895201 30169 tablet_copy_service.cc:140] P 8279b148b2454bfb90afa089240c8086: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 294609dc6da34eefbb57726f025c5619 ({username='oryx', principal='oryx/127.23.115.60@KRBTEST.COM'} at 127.0.0.1:46994)
I20260620 18:38:47.895358 30169 tablet_copy_service.cc:161] P 8279b148b2454bfb90afa089240c8086: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 294609dc6da34eefbb57726f025c5619 at {username='oryx', principal='oryx/127.23.115.60@KRBTEST.COM'} at 127.0.0.1:46994: session id = 294609dc6da34eefbb57726f025c5619-00000000000000000000000000000000
I20260620 18:38:47.898761 30169 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 8279b148b2454bfb90afa089240c8086: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 18:38:47.900677 30711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 18:38:47.907496 30711 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 294609dc6da34eefbb57726f025c5619: tablet copy: Starting download of 0 data blocks...
I20260620 18:38:47.907789 30711 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 294609dc6da34eefbb57726f025c5619: tablet copy: Starting download of 1 WAL segments...
I20260620 18:38:47.910362 30711 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 294609dc6da34eefbb57726f025c5619: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 18:38:47.914716 30169 tablet_copy_service.cc:342] P 8279b148b2454bfb90afa089240c8086: Request end of tablet copy session 294609dc6da34eefbb57726f025c5619-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.23.115.60@KRBTEST.COM'} at 127.0.0.1:46994
I20260620 18:38:47.914868 30169 tablet_copy_service.cc:433] P 8279b148b2454bfb90afa089240c8086: ending tablet copy session 294609dc6da34eefbb57726f025c5619-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 294609dc6da34eefbb57726f025c5619
I20260620 18:38:47.926838 30711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:47.930022 30909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:47.932614 30910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:47.934756 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 30210
I20260620 18:38:47.939502 30711 server_base.cc:1034] running on GCE node
W20260620 18:38:47.940766 30913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:47.941367 30711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:47.942376 30711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:47.943676 30711 hybrid_clock.cc:648] HybridClock initialized: now 1781980727943631 us; error 49 us; skew 500 ppm
I20260620 18:38:47.943989 30711 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:38:47 dist-test-slave-d645 krb5kdc[30104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980727, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.115.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:47.946323 30711 init.cc:377] Logged in from keytab as oryx/127.23.115.60@KRBTEST.COM (short username oryx)
I20260620 18:38:47.947852 30711 webserver.cc:460] Webserver started at http://127.23.115.60:45271/ using document root <none> and password file <none>
I20260620 18:38:47.948135 30711 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:47.948233 30711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:47.950692 30711 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.002s
I20260620 18:38:47.952317 30919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:47.952690 30711 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 18:38:47.952805 30711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/master-2/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/master-2/wal
uuid: "294609dc6da34eefbb57726f025c5619"
format_stamp: "Formatted at 2026-06-20 18:38:47 on dist-test-slave-d645"
server_key: "63333fad221566b28c4a55f7a79c1317"
server_key_iv: "7f1f26e0fd9f5711ae5db60d211d48ed"
server_key_version: "encryptionkey@0"
I20260620 18:38:47.952975 30711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:47.957839 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.115.1:39357
--local_ip_for_outbound_sockets=127.23.115.1
--tserver_master_addrs=127.23.115.62:36693
--webserver_port=43589
--webserver_interface=127.23.115.1
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/krb5kdc/oryx_127.23.115.1.keytab
--principal=oryx/127.23.115.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:34239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.23.115.62:36693,127.23.115.61:45651
--tserver_master_addrs=127.23.115.62:36693,127.23.115.61:45651,127.23.115.60:33051 with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:38:47.972767 30711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:47.976034 30711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:48.007022 30711 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.60:33051
I20260620 18:38:48.007037 30971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.60:33051 every 8 connection(s)
I20260620 18:38:48.007972 30711 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/master-2/data/info.pb
I20260620 18:38:48.013547 30972 sys_catalog.cc:263] Verifying existing consensus state
I20260620 18:38:48.020203 30972 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 294609dc6da34eefbb57726f025c5619: Bootstrap starting.
I20260620 18:38:48.039306 30972 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 294609dc6da34eefbb57726f025c5619: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 18:38:48.039825 30972 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 294609dc6da34eefbb57726f025c5619: Bootstrap complete.
I20260620 18:38:48.040886 30972 raft_consensus.cc:348] T 00000000000000000000000000000000 P 294609dc6da34eefbb57726f025c5619 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8279b148b2454bfb90afa089240c8086" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 36693 } } peers { permanent_uuid: "0d00dab869714576ae199f6caf0c39d8" member_type: VOTER last_known_addr { host: "127.23.115.61" port: 45651 } attrs { promote: false } } peers { permanent_uuid: "294609dc6da34eefbb57726f025c5619" member_type: NON_VOTER last_known_addr { host: "127.23.115.60" port: 33051 } attrs { promote: true } }
I20260620 18:38:48.041158 30972 raft_consensus.cc:729] T 00000000000000000000000000000000 P 294609dc6da34eefbb57726f025c5619 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 294609dc6da34eefbb57726f025c5619, State: Initialized, Role: LEARNER
I20260620 18:38:48.041417 30972 consensus_queue.cc:260] T 00000000000000000000000000000000 P 294609dc6da34eefbb57726f025c5619 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8279b148b2454bfb90afa089240c8086" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 36693 } } peers { permanent_uuid: "0d00dab869714576ae199f6caf0c39d8" member_type: VOTER last_known_addr { host: "127.23.115.61" port: 45651 } attrs { promote: false } } peers { permanent_uuid: "294609dc6da34eefbb57726f025c5619" member_type: NON_VOTER last_known_addr { host: "127.23.115.60" port: 33051 } attrs { promote: true } }
I20260620 18:38:48.042572 30972 sys_catalog.cc:564] T 00000000000000000000000000000000 P 294609dc6da34eefbb57726f025c5619 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:38:48.042261 30975 sys_catalog.cc:455] T 00000000000000000000000000000000 P 294609dc6da34eefbb57726f025c5619 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8279b148b2454bfb90afa089240c8086" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 36693 } } peers { permanent_uuid: "0d00dab869714576ae199f6caf0c39d8" member_type: VOTER last_known_addr { host: "127.23.115.61" port: 45651 } attrs { promote: false } } peers { permanent_uuid: "294609dc6da34eefbb57726f025c5619" member_type: NON_VOTER last_known_addr { host: "127.23.115.60" port: 33051 } attrs { promote: true } } }
I20260620 18:38:48.043184 30975 sys_catalog.cc:458] T 00000000000000000000000000000000 P 294609dc6da34eefbb57726f025c5619 [sys.catalog]: This master's current role is: LEARNER
I20260620 18:38:48.051867 30986 catalog_manager.cc:1259] Loaded cluster ID: f0357000094b4f94b558a6c421dae89d
I20260620 18:38:48.051975 30986 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 294609dc6da34eefbb57726f025c5619: loading cluster ID for follower catalog manager: success
I20260620 18:38:48.054768 30986 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 294609dc6da34eefbb57726f025c5619: acquiring CA information for follower catalog manager: success
I20260620 18:38:48.057350 30986 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 294609dc6da34eefbb57726f025c5619: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 18:38:48.347007 30924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:48.347302 30924 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:48.347433 30924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:48.355352 30924 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:48.355486 30924 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:48.355634 30924 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.115.1
I20260620 18:38:48.366070 30924 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:34239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/krb5kdc/oryx_127.23.115.1.keytab
--principal=oryx/127.23.115.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.115.1:39357
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.115.1
--webserver_port=43589
--tserver_master_addrs=127.23.115.62:36693,127.23.115.61:45651,127.23.115.60:33051
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.115.1
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:38:48.367472 30924 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:48.369333 30924 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:48.383966 30992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:48.384238 30991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:48.384228 30994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:48.385705 30924 server_base.cc:1034] running on GCE node
I20260620 18:38:48.386360 30924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:48.387423 30924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:48.388652 30924 hybrid_clock.cc:648] HybridClock initialized: now 1781980728388606 us; error 46 us; skew 500 ppm
I20260620 18:38:48.388971 30924 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:38:48 dist-test-slave-d645 krb5kdc[30104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980728, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.115.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:48.392613 30924 init.cc:377] Logged in from keytab as oryx/127.23.115.1@KRBTEST.COM (short username oryx)
I20260620 18:38:48.394716 30924 webserver.cc:460] Webserver started at http://127.23.115.1:43589/ using document root <none> and password file <none>
I20260620 18:38:48.395519 30924 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:48.395656 30924 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:48.401012 30924 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 18:38:48.403908 31001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:48.405339 30924 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 18:38:48.405529 30924 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/ts-0/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/ts-0/wal
uuid: "74598bd1acf5422181769e91580ab45a"
format_stamp: "Formatted at 2026-06-20 18:38:45 on dist-test-slave-d645"
server_key: "a69e9c68c12ca5a0ad169079ef965306"
server_key_iv: "962521789ee1c99438cfbc637e7273c4"
server_key_version: "encryptionkey@0"
I20260620 18:38:48.406555 30924 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:48.418435 30924 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:48.423552 30924 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:48.423818 30924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:48.424851 30924 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:38:48.426648 31007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:48.420039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38334 (local address 127.23.115.60:33051)
0620 18:38:48.420639 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:48.420643 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:48.420660 (+ 17us) server_negotiation.cc:407] Connection header received
0620 18:38:48.420732 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:48.420738 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:48.420842 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:48.420966 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:38:48.421345 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:48.422713 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:48.424283 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:48.425446 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:48.425669 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:48.425964 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:48.426130 (+ 166us) server_negotiation.cc:299] Negotiation successful
0620 18:38:48.426272 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":333,"threads_started":1}
I20260620 18:38:48.426712 31006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:48.419869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.60:33051 (local address 127.0.0.1:38334)
0620 18:38:48.420348 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:48.420375 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:48.420512 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:48.420994 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:48.420998 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:48.421020 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:38:48.421173 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:48.421180 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:48.422876 (+ 1696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:48.422880 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:48.424140 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:48.424154 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:48.425748 (+ 1594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:48.425754 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:48.425950 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:48.425979 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 18:38:48.426099 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 18:38:48.426261 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":249,"threads_started":1}
I20260620 18:38:48.426791 30924 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:38:48.426879 30924 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:48.427001 30924 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:38:48.427074 30924 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:48.427670 30947 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 294609dc6da34eefbb57726f025c5619 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260620 18:38:48.428254 30975 sys_catalog.cc:455] T 00000000000000000000000000000000 P 294609dc6da34eefbb57726f025c5619 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8279b148b2454bfb90afa089240c8086. Latest consensus state: current_term: 1 leader_uuid: "8279b148b2454bfb90afa089240c8086" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8279b148b2454bfb90afa089240c8086" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 36693 } } peers { permanent_uuid: "0d00dab869714576ae199f6caf0c39d8" member_type: VOTER last_known_addr { host: "127.23.115.61" port: 45651 } attrs { promote: false } } peers { permanent_uuid: "294609dc6da34eefbb57726f025c5619" member_type: NON_VOTER last_known_addr { host: "127.23.115.60" port: 33051 } attrs { promote: true } } }
I20260620 18:38:48.428526 30975 sys_catalog.cc:458] T 00000000000000000000000000000000 P 294609dc6da34eefbb57726f025c5619 [sys.catalog]: This master's current role is: LEARNER
I20260620 18:38:48.475447 30924 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.1:39357
I20260620 18:38:48.475684 31115 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.1:39357 every 8 connection(s)
I20260620 18:38:48.476904 30924 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/ts-0/data/info.pb
I20260620 18:38:48.485201 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 30924
I20260620 18:38:48.485562 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 30344
Jun 20 18:38:48 dist-test-slave-d645 krb5kdc[30104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980728, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.115.1@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM
I20260620 18:38:48.521116 31120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:48.479891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.1:40753 (local address 127.23.115.62:36693)
0620 18:38:48.495446 (+ 15555us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:48.495453 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:48.499071 (+ 3618us) server_negotiation.cc:407] Connection header received
0620 18:38:48.499777 (+ 706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:48.499784 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:48.499887 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:48.499967 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:48.502594 (+ 2627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:48.503828 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:48.506440 (+ 2612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:48.507059 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:48.507157 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:48.512517 (+ 5360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:48.512536 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:48.512541 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:48.512588 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:48.517047 (+ 4459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:48.518275 (+ 1228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:48.518281 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:48.518283 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:48.518710 (+ 427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:48.519269 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:48.519274 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:48.519278 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:48.519573 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:48.519698 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:48.520446 (+ 748us) server_negotiation.cc:299] Negotiation successful
0620 18:38:48.520669 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15403,"thread_start_us":19042,"threads_started":1}
Jun 20 18:38:48 dist-test-slave-d645 krb5kdc[30104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980728, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.115.1@KRBTEST.COM for oryx/127.23.115.61@KRBTEST.COM
I20260620 18:38:48.522153 31121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:48.480649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:36693 (local address 127.23.115.1:40753)
0620 18:38:48.496119 (+ 15470us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:48.496189 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:48.498863 (+ 2674us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:48.500613 (+ 1750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:48.500627 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:48.501474 (+ 847us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:48.502353 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:48.502378 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:48.504047 (+ 1669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:48.504053 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:48.506256 (+ 2203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:48.506273 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:48.507539 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:48.507545 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:48.507878 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:48.508657 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:48.508688 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:48.512198 (+ 3510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:48.517282 (+ 5084us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:48.517293 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:48.517311 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:48.517953 (+ 642us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:48.518861 (+ 908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:48.518865 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:48.518868 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:48.519151 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:48.519698 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:48.519707 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:48.520065 (+ 358us) client_negotiation.cc:769] Sending connection context
0620 18:38:48.520316 (+ 251us) client_negotiation.cc:240] Negotiation successful
0620 18:38:48.520840 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15119,"thread_start_us":14890,"threads_started":1}
Jun 20 18:38:48 dist-test-slave-d645 krb5kdc[30104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980728, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.115.1@KRBTEST.COM for oryx/127.23.115.60@KRBTEST.COM
I20260620 18:38:48.532610 31124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:48.497124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.1:45297 (local address 127.23.115.61:45651)
0620 18:38:48.499422 (+ 2298us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:48.499428 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:48.510564 (+ 11136us) server_negotiation.cc:407] Connection header received
0620 18:38:48.510808 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:48.510815 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:48.510912 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:48.511015 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:48.513198 (+ 2183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:48.514415 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:48.516748 (+ 2333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:48.517446 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:48.517547 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:48.524984 (+ 7437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:48.524992 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:48.524997 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:48.525045 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:48.529220 (+ 4175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:48.530211 (+ 991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:48.530216 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:48.530220 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:48.530374 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:48.530837 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:48.530842 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:48.530845 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:48.531247 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:48.531383 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:48.531923 (+ 540us) server_negotiation.cc:299] Negotiation successful
0620 18:38:48.532183 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2181,"thread_start_us":2064,"threads_started":1}
I20260620 18:38:48.533638 31125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:48.499426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.61:45651 (local address 127.23.115.1:45297)
0620 18:38:48.510441 (+ 11015us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:48.510466 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:48.510675 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:48.511944 (+ 1269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:48.511950 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:48.512651 (+ 701us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:48.512998 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:48.513010 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:48.514658 (+ 1648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:48.514664 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:48.516523 (+ 1859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:48.516542 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:48.517866 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:48.517872 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:48.518073 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:48.519331 (+ 1258us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:48.519354 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:48.522569 (+ 3215us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:48.529407 (+ 6838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:48.529413 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:48.529417 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:48.530045 (+ 628us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:48.530527 (+ 482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:48.530533 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:48.530537 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:48.530702 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:48.531469 (+ 767us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:48.531474 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:48.531640 (+ 166us) client_negotiation.cc:769] Sending connection context
0620 18:38:48.532811 (+ 1171us) client_negotiation.cc:240] Negotiation successful
0620 18:38:48.533167 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10877,"thread_start_us":8936,"threads_started":1}
Jun 20 18:38:48 dist-test-slave-d645 krb5kdc[30104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980724, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.115.2@KRBTEST.COM
I20260620 18:38:48.538462 31007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:48.509148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.1:39917 (local address 127.23.115.60:33051)
0620 18:38:48.509349 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:48.509355 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:48.522667 (+ 13312us) server_negotiation.cc:407] Connection header received
0620 18:38:48.522796 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:48.522802 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:48.522867 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:48.522979 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:48.523646 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:48.524378 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:48.526402 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:48.527051 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:48.527141 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:48.531847 (+ 4706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:48.531855 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:48.531860 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:48.531902 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:48.535448 (+ 3546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:48.536311 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:48.536317 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:48.536321 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:48.536470 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:48.536909 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:48.536914 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:48.536917 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:48.537251 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:48.537390 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:48.537903 (+ 513us) server_negotiation.cc:299] Negotiation successful
0620 18:38:48.538139 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 18:38:48.539431 31121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:48.509505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.60:33051 (local address 127.23.115.1:39917)
0620 18:38:48.522567 (+ 13062us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:48.522582 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:48.522683 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:48.522996 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:48.522999 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:48.523346 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:48.523487 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:48.523496 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:48.524510 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:48.524514 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:48.526249 (+ 1735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:48.526262 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:48.527380 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:48.527386 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:48.527561 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:48.528187 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:48.528208 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:48.531605 (+ 3397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:48.535619 (+ 4014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:48.535624 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:48.535628 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:48.536184 (+ 556us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:48.536613 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:48.536618 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:48.536621 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:48.536785 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:48.537468 (+ 683us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:48.537473 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:48.537626 (+ 153us) client_negotiation.cc:769] Sending connection context
0620 18:38:48.538722 (+ 1096us) client_negotiation.cc:240] Negotiation successful
0620 18:38:48.539032 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12975,"thread_start_us":29624,"threads_started":1}
I20260620 18:38:48.541304 31118 heartbeater.cc:347] Connected to a master server at 127.23.115.61:45651
I20260620 18:38:48.541363 31117 heartbeater.cc:347] Connected to a master server at 127.23.115.62:36693
I20260620 18:38:48.541615 31118 heartbeater.cc:464] Registering TS with master...
I20260620 18:38:48.541622 31117 heartbeater.cc:464] Registering TS with master...
I20260620 18:38:48.542688 31118 heartbeater.cc:511] Master 127.23.115.61:45651 requested a full tablet report, sending...
I20260620 18:38:48.542709 31117 heartbeater.cc:511] Master 127.23.115.62:36693 requested a full tablet report, sending...
I20260620 18:38:48.544896 31123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:48.486907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38108 (local address 127.23.115.2:33167)
0620 18:38:48.518854 (+ 31947us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:48.518863 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:48.520795 (+ 1932us) server_negotiation.cc:407] Connection header received
0620 18:38:48.520876 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:48.520885 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:48.521149 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:48.521609 (+ 460us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:48.524845 (+ 3236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:48.526346 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:48.528382 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:48.529355 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:48.529489 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:48.537234 (+ 7745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:48.537252 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:48.537271 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:48.537312 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:48.541257 (+ 3945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:48.542238 (+ 981us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:48.542247 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:48.542250 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:48.542452 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:48.542950 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:48.542956 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:48.542960 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:48.543412 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:48.543716 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:48.544211 (+ 495us) server_negotiation.cc:299] Negotiation successful
0620 18:38:48.544520 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31432,"thread_start_us":31507,"threads_started":1}
I20260620 18:38:48.545234 30748 ts_manager.cc:194] Registered new tserver with Master: 74598bd1acf5422181769e91580ab45a (127.23.115.1:39357)
I20260620 18:38:48.546003 31116 heartbeater.cc:347] Connected to a master server at 127.23.115.60:33051
I20260620 18:38:48.546151 31116 heartbeater.cc:464] Registering TS with master...
I20260620 18:38:48.546361 30467 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:38:48.546633 31116 heartbeater.cc:511] Master 127.23.115.60:33051 requested a full tablet report, sending...
I20260620 18:38:48.546797 30149 ts_manager.cc:194] Re-registered known tserver with Master: 74598bd1acf5422181769e91580ab45a (127.23.115.1:39357)
I20260620 18:38:48.547789 30149 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.115.1@KRBTEST.COM'} at 127.23.115.1:40753
I20260620 18:38:48.548532 30937 ts_manager.cc:194] Registered new tserver with Master: 74598bd1acf5422181769e91580ab45a (127.23.115.1:39357)
I20260620 18:38:48.631807 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 30344
I20260620 18:38:48.645411 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.115.2:33167
--local_ip_for_outbound_sockets=127.23.115.2
--tserver_master_addrs=127.23.115.62:36693
--webserver_port=36175
--webserver_interface=127.23.115.2
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/krb5kdc/oryx_127.23.115.2.keytab
--principal=oryx/127.23.115.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:34239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.23.115.62:36693,127.23.115.61:45651
--tserver_master_addrs=127.23.115.62:36693,127.23.115.61:45651,127.23.115.60:33051 with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:38:48.979828 31128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:48.980093 31128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:48.980185 31128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:48.987573 31128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:48.987691 31128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:48.987823 31128 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.115.2
I20260620 18:38:48.997617 31128 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:34239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/krb5kdc/oryx_127.23.115.2.keytab
--principal=oryx/127.23.115.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.115.2:33167
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.115.2
--webserver_port=36175
--tserver_master_addrs=127.23.115.62:36693,127.23.115.61:45651,127.23.115.60:33051
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.115.2
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:38:48.998880 31128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:49.000190 31128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 18:38:49.001636 30895 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8279b148b2454bfb90afa089240c8086 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8279b148b2454bfb90afa089240c8086" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8279b148b2454bfb90afa089240c8086" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 36693 } } peers { permanent_uuid: "0d00dab869714576ae199f6caf0c39d8" member_type: VOTER last_known_addr { host: "127.23.115.61" port: 45651 } attrs { promote: false } } peers { permanent_uuid: "294609dc6da34eefbb57726f025c5619" member_type: NON_VOTER last_known_addr { host: "127.23.115.60" port: 33051 } attrs { promote: true } } }
I20260620 18:38:49.001932 30895 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8279b148b2454bfb90afa089240c8086 [sys.catalog]: This master's current role is: LEADER
I20260620 18:38:49.003325 30895 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 8279b148b2454bfb90afa089240c8086: attempting to promote NON_VOTER 294609dc6da34eefbb57726f025c5619 to VOTER
I20260620 18:38:49.004061 30895 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8279b148b2454bfb90afa089240c8086 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8279b148b2454bfb90afa089240c8086" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 36693 } } peers { permanent_uuid: "0d00dab869714576ae199f6caf0c39d8" member_type: VOTER last_known_addr { host: "127.23.115.61" port: 45651 } attrs { promote: false } } peers { permanent_uuid: "294609dc6da34eefbb57726f025c5619" member_type: VOTER last_known_addr { host: "127.23.115.60" port: 33051 } attrs { promote: false } }
I20260620 18:38:49.006438 30758 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0d00dab869714576ae199f6caf0c39d8 [term 1 FOLLOWER]: Refusing update from remote peer 8279b148b2454bfb90afa089240c8086: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260620 18:38:49.006482 30947 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 294609dc6da34eefbb57726f025c5619 [term 1 LEARNER]: Refusing update from remote peer 8279b148b2454bfb90afa089240c8086: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260620 18:38:49.007170 30895 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8279b148b2454bfb90afa089240c8086 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d00dab869714576ae199f6caf0c39d8" member_type: VOTER last_known_addr { host: "127.23.115.61" port: 45651 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260620 18:38:49.007555 31133 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8279b148b2454bfb90afa089240c8086 [LEADER]: Connected to new peer: Peer: permanent_uuid: "294609dc6da34eefbb57726f025c5619" member_type: VOTER last_known_addr { host: "127.23.115.60" port: 33051 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260620 18:38:49.014061 31133 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8279b148b2454bfb90afa089240c8086 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 294609dc6da34eefbb57726f025c5619 (127.23.115.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8279b148b2454bfb90afa089240c8086" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 36693 } } peers { permanent_uuid: "0d00dab869714576ae199f6caf0c39d8" member_type: VOTER last_known_addr { host: "127.23.115.61" port: 45651 } attrs { promote: false } } peers { permanent_uuid: "294609dc6da34eefbb57726f025c5619" member_type: VOTER last_known_addr { host: "127.23.115.60" port: 33051 } attrs { promote: false } } }
I20260620 18:38:49.016243 30758 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0d00dab869714576ae199f6caf0c39d8 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 294609dc6da34eefbb57726f025c5619 (127.23.115.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8279b148b2454bfb90afa089240c8086" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 36693 } } peers { permanent_uuid: "0d00dab869714576ae199f6caf0c39d8" member_type: VOTER last_known_addr { host: "127.23.115.61" port: 45651 } attrs { promote: false } } peers { permanent_uuid: "294609dc6da34eefbb57726f025c5619" member_type: VOTER last_known_addr { host: "127.23.115.60" port: 33051 } attrs { promote: false } } }
I20260620 18:38:49.016489 30947 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 294609dc6da34eefbb57726f025c5619 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 294609dc6da34eefbb57726f025c5619 (127.23.115.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8279b148b2454bfb90afa089240c8086" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 36693 } } peers { permanent_uuid: "0d00dab869714576ae199f6caf0c39d8" member_type: VOTER last_known_addr { host: "127.23.115.61" port: 45651 } attrs { promote: false } } peers { permanent_uuid: "294609dc6da34eefbb57726f025c5619" member_type: VOTER last_known_addr { host: "127.23.115.60" port: 33051 } attrs { promote: false } } }
I20260620 18:38:49.021677 31141 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d00dab869714576ae199f6caf0c39d8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8279b148b2454bfb90afa089240c8086" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8279b148b2454bfb90afa089240c8086" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 36693 } } peers { permanent_uuid: "0d00dab869714576ae199f6caf0c39d8" member_type: VOTER last_known_addr { host: "127.23.115.61" port: 45651 } attrs { promote: false } } peers { permanent_uuid: "294609dc6da34eefbb57726f025c5619" member_type: VOTER last_known_addr { host: "127.23.115.60" port: 33051 } attrs { promote: false } } }
I20260620 18:38:49.022130 31141 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d00dab869714576ae199f6caf0c39d8 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:38:49.023483 31132 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8279b148b2454bfb90afa089240c8086 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "8279b148b2454bfb90afa089240c8086" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8279b148b2454bfb90afa089240c8086" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 36693 } } peers { permanent_uuid: "0d00dab869714576ae199f6caf0c39d8" member_type: VOTER last_known_addr { host: "127.23.115.61" port: 45651 } attrs { promote: false } } peers { permanent_uuid: "294609dc6da34eefbb57726f025c5619" member_type: VOTER last_known_addr { host: "127.23.115.60" port: 33051 } attrs { promote: false } } }
I20260620 18:38:49.023940 31132 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8279b148b2454bfb90afa089240c8086 [sys.catalog]: This master's current role is: LEADER
I20260620 18:38:49.023634 31133 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8279b148b2454bfb90afa089240c8086 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8279b148b2454bfb90afa089240c8086" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8279b148b2454bfb90afa089240c8086" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 36693 } } peers { permanent_uuid: "0d00dab869714576ae199f6caf0c39d8" member_type: VOTER last_known_addr { host: "127.23.115.61" port: 45651 } attrs { promote: false } } peers { permanent_uuid: "294609dc6da34eefbb57726f025c5619" member_type: VOTER last_known_addr { host: "127.23.115.60" port: 33051 } attrs { promote: false } } }
I20260620 18:38:49.024457 31133 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8279b148b2454bfb90afa089240c8086 [sys.catalog]: This master's current role is: LEADER
I20260620 18:38:49.024533 31132 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8279b148b2454bfb90afa089240c8086 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8279b148b2454bfb90afa089240c8086" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8279b148b2454bfb90afa089240c8086" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 36693 } } peers { permanent_uuid: "0d00dab869714576ae199f6caf0c39d8" member_type: VOTER last_known_addr { host: "127.23.115.61" port: 45651 } attrs { promote: false } } peers { permanent_uuid: "294609dc6da34eefbb57726f025c5619" member_type: VOTER last_known_addr { host: "127.23.115.60" port: 33051 } attrs { promote: false } } }
I20260620 18:38:49.024802 31132 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8279b148b2454bfb90afa089240c8086 [sys.catalog]: This master's current role is: LEADER
W20260620 18:38:49.029176 31140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:49.029206 31138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:49.031239 31142 sys_catalog.cc:455] T 00000000000000000000000000000000 P 294609dc6da34eefbb57726f025c5619 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8279b148b2454bfb90afa089240c8086" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8279b148b2454bfb90afa089240c8086" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 36693 } } peers { permanent_uuid: "0d00dab869714576ae199f6caf0c39d8" member_type: VOTER last_known_addr { host: "127.23.115.61" port: 45651 } attrs { promote: false } } peers { permanent_uuid: "294609dc6da34eefbb57726f025c5619" member_type: VOTER last_known_addr { host: "127.23.115.60" port: 33051 } attrs { promote: false } } }
I20260620 18:38:49.032110 31142 sys_catalog.cc:458] T 00000000000000000000000000000000 P 294609dc6da34eefbb57726f025c5619 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 18:38:49.032840 31144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:49.033730 31128 server_base.cc:1034] running on GCE node
I20260620 18:38:49.034519 31128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:49.035631 31128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:49.036861 31128 hybrid_clock.cc:648] HybridClock initialized: now 1781980729036815 us; error 38 us; skew 500 ppm
I20260620 18:38:49.037214 31128 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:38:49 dist-test-slave-d645 krb5kdc[30104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980729, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.115.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:49.040712 31128 init.cc:377] Logged in from keytab as oryx/127.23.115.2@KRBTEST.COM (short username oryx)
I20260620 18:38:49.042333 31128 webserver.cc:460] Webserver started at http://127.23.115.2:36175/ using document root <none> and password file <none>
I20260620 18:38:49.043083 31128 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:49.043200 31128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:49.048813 31128 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 18:38:49.051613 31151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:49.053167 31128 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:38:49.053340 31128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/ts-1/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/ts-1/wal
uuid: "7b659e72cc154daf8c7b6de5cc3d0c29"
format_stamp: "Formatted at 2026-06-20 18:38:45 on dist-test-slave-d645"
server_key: "7e516ef6f859822c2c3689cf97877175"
server_key_iv: "723d0e5857a4112414f630619f06e2a3"
server_key_version: "encryptionkey@0"
I20260620 18:38:49.054407 31128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:49.075603 31128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:49.079833 31128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:49.080104 31128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:49.081089 31128 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:38:49.082459 31128 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:38:49.082553 31128 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:49.082664 31128 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:38:49.082746 31128 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:49.114221 31128 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.2:33167
I20260620 18:38:49.114275 31263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.2:33167 every 8 connection(s)
I20260620 18:38:49.115665 31128 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/ts-1/data/info.pb
I20260620 18:38:49.125851 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 31128
I20260620 18:38:49.126195 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 30477
Jun 20 18:38:49 dist-test-slave-d645 krb5kdc[30104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980729, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.115.2@KRBTEST.COM for oryx/127.23.115.60@KRBTEST.COM
Jun 20 18:38:49 dist-test-slave-d645 krb5kdc[30104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980729, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.115.2@KRBTEST.COM for oryx/127.23.115.61@KRBTEST.COM
Jun 20 18:38:49 dist-test-slave-d645 krb5kdc[30104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980724, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.115.3@KRBTEST.COM
I20260620 18:38:49.145644 31268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:49.118834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.2:36725 (local address 127.23.115.60:33051)
0620 18:38:49.119420 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:49.119426 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:49.120492 (+ 1066us) server_negotiation.cc:407] Connection header received
0620 18:38:49.128434 (+ 7942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:49.128442 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:49.128539 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:49.130000 (+ 1461us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:49.130184 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:49.131252 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:49.132982 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:49.133809 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:49.133999 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:49.138530 (+ 4531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:49.138538 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:49.138541 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:49.138574 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:49.141462 (+ 2888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:49.142560 (+ 1098us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:49.142565 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:49.142568 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:49.142740 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:49.143378 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:49.143382 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:49.143385 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:49.143758 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:49.143915 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:49.144978 (+ 1063us) server_negotiation.cc:299] Negotiation successful
0620 18:38:49.145214 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":285,"threads_started":1}
I20260620 18:38:49.146867 31269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:49.119464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.60:33051 (local address 127.23.115.2:36725)
0620 18:38:49.120323 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:49.120373 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:49.123166 (+ 2793us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:49.129002 (+ 5836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:49.129007 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:49.129466 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:49.129781 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:49.129796 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:49.131372 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:49.131376 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:49.132817 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:49.132832 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:49.134052 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:49.134060 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:49.134252 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:49.134845 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:49.134864 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:49.138288 (+ 3424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:49.141640 (+ 3352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:49.141649 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:49.141668 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:49.142399 (+ 731us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:49.142873 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:49.142878 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:49.142882 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:49.143189 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:49.143949 (+ 760us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:49.143959 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:49.144435 (+ 476us) client_negotiation.cc:769] Sending connection context
0620 18:38:49.144833 (+ 398us) client_negotiation.cc:240] Negotiation successful
0620 18:38:49.145411 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"spinlock_wait_cycles":30464,"thread_start_us":337,"threads_started":1}
I20260620 18:38:49.147547 31273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:49.125052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.61:45651 (local address 127.23.115.2:58459)
0620 18:38:49.125570 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:49.125591 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:49.125777 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:49.126815 (+ 1038us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:49.126823 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:49.127641 (+ 818us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:49.128470 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:49.128496 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:49.129758 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:49.129762 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:49.131161 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:49.131176 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:49.133087 (+ 1911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:49.133094 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:49.133371 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:49.134624 (+ 1253us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:49.134685 (+ 61us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:49.138288 (+ 3603us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:49.142873 (+ 4585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:49.142878 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:49.142882 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:49.143383 (+ 501us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:49.143818 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:49.143823 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:49.143827 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:49.143987 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:49.146342 (+ 2355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:49.146347 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:49.146491 (+ 144us) client_negotiation.cc:769] Sending connection context
0620 18:38:49.146664 (+ 173us) client_negotiation.cc:240] Negotiation successful
0620 18:38:49.146969 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":286,"threads_started":1}
I20260620 18:38:49.147756 31272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:49.124257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.2:58459 (local address 127.23.115.61:45651)
0620 18:38:49.124719 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:49.124724 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:49.125663 (+ 939us) server_negotiation.cc:407] Connection header received
0620 18:38:49.126056 (+ 393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:49.126062 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:49.126238 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:49.126324 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:49.128681 (+ 2357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:49.129585 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:49.131314 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:49.132056 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:49.132199 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:49.138554 (+ 6355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:49.138563 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:49.138569 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:49.138643 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:49.142718 (+ 4075us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:49.143525 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:49.143532 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:49.143536 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:49.143698 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:49.144106 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:49.144111 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:49.144114 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:49.144486 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:49.144629 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:49.146837 (+ 2208us) server_negotiation.cc:299] Negotiation successful
0620 18:38:49.147239 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":237,"threads_started":1}
I20260620 18:38:49.148664 31264 heartbeater.cc:347] Connected to a master server at 127.23.115.60:33051
I20260620 18:38:49.149013 31264 heartbeater.cc:464] Registering TS with master...
I20260620 18:38:49.149973 31264 heartbeater.cc:511] Master 127.23.115.60:33051 requested a full tablet report, sending...
I20260620 18:38:49.148689 31266 heartbeater.cc:347] Connected to a master server at 127.23.115.61:45651
I20260620 18:38:49.151065 31266 heartbeater.cc:464] Registering TS with master...
I20260620 18:38:49.152103 31266 heartbeater.cc:511] Master 127.23.115.61:45651 requested a full tablet report, sending...
Jun 20 18:38:49 dist-test-slave-d645 krb5kdc[30104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980729, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.115.2@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM
I20260620 18:38:49.153002 31275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:49.127746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46064 (local address 127.23.115.3:44857)
0620 18:38:49.128416 (+ 670us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:49.128424 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:49.128452 (+ 28us) server_negotiation.cc:407] Connection header received
0620 18:38:49.128514 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:49.128521 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:49.128782 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:49.128929 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:49.131043 (+ 2114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:49.132867 (+ 1824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:49.134924 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:49.135966 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:49.136128 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:49.146140 (+ 10012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:49.146158 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:49.146176 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:49.146220 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:49.149998 (+ 3778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:49.150747 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:49.150754 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:49.150756 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:49.150884 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:49.151328 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:49.151333 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:49.151337 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:49.151663 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:49.151934 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:49.152363 (+ 429us) server_negotiation.cc:299] Negotiation successful
0620 18:38:49.152628 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":504,"thread_start_us":750,"threads_started":1}
I20260620 18:38:49.153573 30748 ts_manager.cc:194] Registered new tserver with Master: 7b659e72cc154daf8c7b6de5cc3d0c29 (127.23.115.2:33167)
I20260620 18:38:49.153559 30937 ts_manager.cc:194] Registered new tserver with Master: 7b659e72cc154daf8c7b6de5cc3d0c29 (127.23.115.2:33167)
I20260620 18:38:49.154227 30600 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:38:49.161326 31271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:49.121088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.2:50215 (local address 127.23.115.62:36693)
0620 18:38:49.140494 (+ 19406us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:49.140502 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:49.140536 (+ 34us) server_negotiation.cc:407] Connection header received
0620 18:38:49.140606 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:49.140611 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:49.140715 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:49.140847 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:49.142076 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:49.144251 (+ 2175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:49.148233 (+ 3982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:49.148890 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:49.149020 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:49.154743 (+ 5723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:49.154752 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:49.154757 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:49.154819 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:49.158382 (+ 3563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:49.159319 (+ 937us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:49.159323 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:49.159326 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:49.159461 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:49.159935 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:49.159939 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:49.159941 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:49.160213 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:49.160357 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:49.160852 (+ 495us) server_negotiation.cc:299] Negotiation successful
0620 18:38:49.161017 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19245,"thread_start_us":19882,"threads_started":1}
I20260620 18:38:49.161540 31270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:49.120921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:36693 (local address 127.23.115.2:50215)
0620 18:38:49.124009 (+ 3088us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:49.124027 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:49.124152 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:49.140909 (+ 16757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:49.140916 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:49.141451 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:49.141887 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:49.141904 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:49.144396 (+ 2492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:49.144401 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:49.145935 (+ 1534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:49.145950 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:49.149138 (+ 3188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:49.149146 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:49.149355 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:49.149946 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:49.149977 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:49.154500 (+ 4523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:49.158546 (+ 4046us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:49.158553 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:49.158558 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:49.159138 (+ 580us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:49.159594 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:49.159600 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:49.159605 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:49.159801 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:49.160389 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:49.160394 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:49.160579 (+ 185us) client_negotiation.cc:769] Sending connection context
0620 18:38:49.160733 (+ 154us) client_negotiation.cc:240] Negotiation successful
0620 18:38:49.161060 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2952,"thread_start_us":2777,"threads_started":1}
I20260620 18:38:49.162420 31265 heartbeater.cc:347] Connected to a master server at 127.23.115.62:36693
I20260620 18:38:49.162559 31265 heartbeater.cc:464] Registering TS with master...
I20260620 18:38:49.162962 31265 heartbeater.cc:511] Master 127.23.115.62:36693 requested a full tablet report, sending...
I20260620 18:38:49.164045 30149 ts_manager.cc:194] Re-registered known tserver with Master: 7b659e72cc154daf8c7b6de5cc3d0c29 (127.23.115.2:33167)
I20260620 18:38:49.165215 30149 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.115.2@KRBTEST.COM'} at 127.23.115.2:50215
I20260620 18:38:49.240818 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 30477
I20260620 18:38:49.255334 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.115.3:44857
--local_ip_for_outbound_sockets=127.23.115.3
--tserver_master_addrs=127.23.115.62:36693
--webserver_port=35111
--webserver_interface=127.23.115.3
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/krb5kdc/oryx_127.23.115.3.keytab
--principal=oryx/127.23.115.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:34239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.23.115.62:36693,127.23.115.61:45651
--tserver_master_addrs=127.23.115.62:36693,127.23.115.61:45651,127.23.115.60:33051 with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:38:49.550396 31117 heartbeater.cc:503] Master 127.23.115.62:36693 was elected leader, sending a full tablet report...
W20260620 18:38:49.591329 31277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:49.591589 31277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:49.591681 31277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:49.598784 31277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:49.598891 31277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:49.599045 31277 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.115.3
I20260620 18:38:49.608546 31277 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:34239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/krb5kdc/oryx_127.23.115.3.keytab
--principal=oryx/127.23.115.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.115.3:44857
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.115.3
--webserver_port=35111
--tserver_master_addrs=127.23.115.62:36693,127.23.115.61:45651,127.23.115.60:33051
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.115.3
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:38:49.609935 31277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:49.611403 31277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:49.624395 31282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:49.624369 31283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:49.624341 31285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:49.625514 31277 server_base.cc:1034] running on GCE node
I20260620 18:38:49.626117 31277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:49.627137 31277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:49.628363 31277 hybrid_clock.cc:648] HybridClock initialized: now 1781980729628317 us; error 36 us; skew 500 ppm
I20260620 18:38:49.628669 31277 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:38:49 dist-test-slave-d645 krb5kdc[30104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980729, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.115.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:49.632005 31277 init.cc:377] Logged in from keytab as oryx/127.23.115.3@KRBTEST.COM (short username oryx)
I20260620 18:38:49.633486 31277 webserver.cc:460] Webserver started at http://127.23.115.3:35111/ using document root <none> and password file <none>
I20260620 18:38:49.634155 31277 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:49.634276 31277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:49.639633 31277 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 18:38:49.642360 31292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:49.643944 31277 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 18:38:49.644120 31277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/ts-2/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/ts-2/wal
uuid: "b8959fed0d23477fab84fd0a790447ce"
format_stamp: "Formatted at 2026-06-20 18:38:46 on dist-test-slave-d645"
server_key: "3d4b70bd1a5b2a78f305ccd161da052b"
server_key_iv: "4c8563e2882971ba9d9c39d82aee57e5"
server_key_version: "encryptionkey@0"
I20260620 18:38:49.645105 31277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:49.662851 31277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:49.666577 31277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:49.666832 31277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:49.667783 31277 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:38:49.669149 31277 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:38:49.669238 31277 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:49.669348 31277 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:38:49.669411 31277 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:49.701047 31277 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.3:44857
I20260620 18:38:49.701135 31404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.3:44857 every 8 connection(s)
I20260620 18:38:49.702979 31277 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/ts-2/data/info.pb
I20260620 18:38:49.711187 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 31277
Jun 20 18:38:49 dist-test-slave-d645 krb5kdc[30104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980729, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.115.3@KRBTEST.COM for oryx/127.23.115.61@KRBTEST.COM
Jun 20 18:38:49 dist-test-slave-d645 krb5kdc[30104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980729, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.115.3@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM
Jun 20 18:38:49 dist-test-slave-d645 krb5kdc[30104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980729, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.115.3@KRBTEST.COM for oryx/127.23.115.60@KRBTEST.COM
I20260620 18:38:49.731827 31413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:49.708855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.3:55121 (local address 127.23.115.61:45651)
0620 18:38:49.709306 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:49.709312 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:49.709816 (+ 504us) server_negotiation.cc:407] Connection header received
0620 18:38:49.710563 (+ 747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:49.710569 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:49.710649 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:49.710758 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:49.713245 (+ 2487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:49.714414 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:49.717586 (+ 3172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:49.718225 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:49.718324 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:49.723934 (+ 5610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:49.723942 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:49.723948 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:49.723995 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:49.727553 (+ 3558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:49.728637 (+ 1084us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:49.728645 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:49.728649 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:49.728834 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:49.729437 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:49.729443 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:49.729447 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:49.729838 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:49.729971 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:49.731075 (+ 1104us) server_negotiation.cc:299] Negotiation successful
0620 18:38:49.731314 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":403,"threads_started":1}
I20260620 18:38:49.732851 31412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:49.708105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.3:42701 (local address 127.23.115.62:36693)
0620 18:38:49.708974 (+ 869us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:49.708980 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:49.708999 (+ 19us) server_negotiation.cc:407] Connection header received
0620 18:38:49.710256 (+ 1257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:49.710262 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:49.710350 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:49.710450 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:49.713245 (+ 2795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:49.714414 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:49.717586 (+ 3172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:49.718225 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:49.718338 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:49.725354 (+ 7016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:49.725364 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:49.725369 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:49.725456 (+ 87us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:49.728922 (+ 3466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:49.729775 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:49.729781 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:49.729784 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:49.729948 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:49.730395 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:49.730400 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:49.730403 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:49.731153 (+ 750us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:49.731303 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:49.732273 (+ 970us) server_negotiation.cc:299] Negotiation successful
0620 18:38:49.732507 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":757,"thread_start_us":650,"threads_started":1}
I20260620 18:38:49.734042 31409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:49.706037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.3:47623 (local address 127.23.115.60:33051)
0620 18:38:49.706636 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:49.706641 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:49.707554 (+ 913us) server_negotiation.cc:407] Connection header received
0620 18:38:49.710130 (+ 2576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:49.710137 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:49.710240 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:49.710357 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:49.713241 (+ 2884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:49.714470 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:49.717640 (+ 3170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:49.718320 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:49.718412 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:49.725302 (+ 6890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:49.725311 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:49.725317 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:49.725364 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:49.730269 (+ 4905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:49.731470 (+ 1201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:49.731477 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:49.731481 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:49.731626 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:49.732223 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:49.732228 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:49.732232 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:49.732637 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:49.732782 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:49.733356 (+ 574us) server_negotiation.cc:299] Negotiation successful
0620 18:38:49.733600 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":318,"threads_started":1}
I20260620 18:38:49.734193 31414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:49.709144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.61:45651 (local address 127.23.115.3:55121)
0620 18:38:49.709713 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:49.709734 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:49.709852 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:49.710967 (+ 1115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:49.710972 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:49.711840 (+ 868us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:49.713020 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:49.713048 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:49.714653 (+ 1605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:49.714658 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:49.717411 (+ 2753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:49.717420 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:49.718685 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:49.718692 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:49.719158 (+ 466us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:49.720276 (+ 1118us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:49.720296 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:49.723623 (+ 3327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:49.727747 (+ 4124us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:49.727757 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:49.727778 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:49.728419 (+ 641us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:49.728996 (+ 577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:49.729001 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:49.729005 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:49.729287 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:49.730089 (+ 802us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:49.730097 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:49.730555 (+ 458us) client_negotiation.cc:769] Sending connection context
0620 18:38:49.732095 (+ 1540us) client_negotiation.cc:240] Negotiation successful
0620 18:38:49.732659 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"mutex_wait_us":662,"spinlock_wait_cycles":1792,"thread_start_us":304,"threads_started":1}
I20260620 18:38:49.734602 31411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:49.707907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:36693 (local address 127.23.115.3:42701)
0620 18:38:49.708524 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:49.708560 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:49.709856 (+ 1296us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:49.710707 (+ 851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:49.710720 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:49.711840 (+ 1120us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:49.713020 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:49.713048 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:49.714637 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:49.714644 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:49.717400 (+ 2756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:49.717416 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:49.718685 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:49.718692 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:49.719152 (+ 460us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:49.720086 (+ 934us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:49.720123 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:49.724279 (+ 4156us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:49.729076 (+ 4797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:49.729080 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:49.729084 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:49.729627 (+ 543us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:49.730080 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:49.730085 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:49.730088 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:49.730254 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:49.731369 (+ 1115us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:49.731374 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:49.731614 (+ 240us) client_negotiation.cc:769] Sending connection context
0620 18:38:49.731833 (+ 219us) client_negotiation.cc:240] Negotiation successful
0620 18:38:49.733487 (+ 1654us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"mutex_wait_us":504,"spinlock_wait_cycles":30336,"thread_start_us":281,"threads_started":1}
I20260620 18:38:49.735527 31410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:49.706609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.60:33051 (local address 127.23.115.3:47623)
0620 18:38:49.707402 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:49.707447 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:49.709849 (+ 2402us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:49.710708 (+ 859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:49.710719 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:49.711998 (+ 1279us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:49.713032 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:49.713057 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:49.714671 (+ 1614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:49.714677 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:49.717403 (+ 2726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:49.717420 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:49.718797 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:49.718808 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:49.719140 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:49.720451 (+ 1311us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:49.720476 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:49.725018 (+ 4542us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:49.730778 (+ 5760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:49.730784 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:49.730788 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:49.731323 (+ 535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:49.731778 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:49.731783 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:49.731787 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:49.732055 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:49.732872 (+ 817us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:49.732877 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:49.733077 (+ 200us) client_negotiation.cc:769] Sending connection context
0620 18:38:49.734314 (+ 1237us) client_negotiation.cc:240] Negotiation successful
0620 18:38:49.734664 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"mutex_wait_us":372,"spinlock_wait_cycles":14080,"thread_start_us":336,"threads_started":1}
I20260620 18:38:49.736151 31407 heartbeater.cc:347] Connected to a master server at 127.23.115.61:45651
I20260620 18:38:49.736485 31406 heartbeater.cc:347] Connected to a master server at 127.23.115.62:36693
I20260620 18:38:49.736526 31407 heartbeater.cc:464] Registering TS with master...
I20260620 18:38:49.736611 31406 heartbeater.cc:464] Registering TS with master...
I20260620 18:38:49.737027 31405 heartbeater.cc:347] Connected to a master server at 127.23.115.60:33051
I20260620 18:38:49.737154 31405 heartbeater.cc:464] Registering TS with master...
I20260620 18:38:49.738044 31405 heartbeater.cc:511] Master 127.23.115.60:33051 requested a full tablet report, sending...
I20260620 18:38:49.738188 31406 heartbeater.cc:511] Master 127.23.115.62:36693 requested a full tablet report, sending...
I20260620 18:38:49.738095 31407 heartbeater.cc:511] Master 127.23.115.61:45651 requested a full tablet report, sending...
I20260620 18:38:49.740123 30937 ts_manager.cc:194] Registered new tserver with Master: b8959fed0d23477fab84fd0a790447ce (127.23.115.3:44857)
I20260620 18:38:49.740191 30748 ts_manager.cc:194] Registered new tserver with Master: b8959fed0d23477fab84fd0a790447ce (127.23.115.3:44857)
I20260620 18:38:49.740747 30149 ts_manager.cc:194] Re-registered known tserver with Master: b8959fed0d23477fab84fd0a790447ce (127.23.115.3:44857)
I20260620 18:38:49.741624 30149 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.115.3@KRBTEST.COM'} at 127.23.115.3:42701
I20260620 18:38:49.761981 31413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:49.745479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34580 (local address 127.23.115.61:45651)
0620 18:38:49.745681 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:49.745688 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:49.746294 (+ 606us) server_negotiation.cc:407] Connection header received
0620 18:38:49.746453 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:49.746459 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:49.746547 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:49.746647 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:49.748890 (+ 2243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:49.749837 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:49.751778 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:49.752405 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:49.752520 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:49.755718 (+ 3198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:49.755745 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:49.755750 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:49.755791 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:49.758967 (+ 3176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:49.759772 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:49.759778 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:49.759781 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:49.759932 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:49.760440 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:49.760445 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:49.760448 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:49.760844 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:49.761105 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:49.761437 (+ 332us) server_negotiation.cc:299] Negotiation successful
0620 18:38:49.761657 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 18:38:49.762694 31412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:49.744467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47002 (local address 127.23.115.62:36693)
0620 18:38:49.744662 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:49.744669 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:49.745293 (+ 624us) server_negotiation.cc:407] Connection header received
0620 18:38:49.745531 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:49.745536 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:49.745678 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:49.745760 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:49.748478 (+ 2718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:49.749621 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:49.751381 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:49.752077 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:49.752221 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:49.756356 (+ 4135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:49.756363 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:49.756368 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:49.756409 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:49.759686 (+ 3277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:49.760475 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:49.760480 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:49.760484 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:49.760619 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:49.761040 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:49.761045 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:49.761049 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:49.761399 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:49.761618 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:49.762164 (+ 546us) server_negotiation.cc:299] Negotiation successful
0620 18:38:49.762410 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 18:38:49.764207 31409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:49.746390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38348 (local address 127.23.115.60:33051)
0620 18:38:49.746581 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:49.746585 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:49.747676 (+ 1091us) server_negotiation.cc:407] Connection header received
0620 18:38:49.747860 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:49.747866 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:49.747954 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:49.748054 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:49.750492 (+ 2438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:49.751620 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:49.753326 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:49.754033 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:49.754144 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:49.757828 (+ 3684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:49.757835 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:49.757840 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:49.757882 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:49.761106 (+ 3224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:49.761943 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:49.761949 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:49.761952 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:49.762119 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:49.762534 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:49.762539 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:49.762542 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:49.762894 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:49.763175 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:49.763682 (+ 507us) server_negotiation.cc:299] Negotiation successful
0620 18:38:49.763906 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 18:38:49.767666 30149 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47002:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:38:49.774469 30149 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:38:49.803260 31431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:49.791918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38116 (local address 127.23.115.2:33167)
0620 18:38:49.796769 (+ 4851us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:49.796776 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:49.796799 (+ 23us) server_negotiation.cc:407] Connection header received
0620 18:38:49.796883 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:49.796890 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:49.797077 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:49.797229 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:38:49.797728 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:49.798875 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:49.800870 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:49.802190 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:49.802302 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:49.802520 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:49.802756 (+ 236us) server_negotiation.cc:299] Negotiation successful
0620 18:38:49.802980 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4713,"thread_start_us":4593,"threads_started":1}
I20260620 18:38:49.803310 31430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:49.791689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.2:33167 (local address 127.0.0.1:38116)
0620 18:38:49.792309 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:49.792332 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:49.792465 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:49.797265 (+ 4800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:49.797268 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:49.797289 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:38:49.797537 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:49.797545 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:49.799011 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:49.799017 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:49.800721 (+ 1704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:49.800733 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:49.802389 (+ 1656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:49.802395 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:49.802584 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:49.802612 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 18:38:49.802697 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 18:38:49.802843 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"mutex_wait_us":90,"thread_start_us":331,"threads_started":1}
I20260620 18:38:49.807075 31433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:49.793088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.3:44857 (local address 127.0.0.1:46070)
0620 18:38:49.793976 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:49.793992 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:49.794099 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:49.798122 (+ 4023us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:49.798126 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:49.798154 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:38:49.798516 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:49.798528 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:49.801473 (+ 2945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:49.801479 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:49.804071 (+ 2592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:49.804087 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:49.806254 (+ 2167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:49.806260 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:49.806447 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:49.806482 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 18:38:49.806568 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 18:38:49.806702 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":775,"thread_start_us":2942,"threads_started":1}
I20260620 18:38:49.807130 31434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:49.793497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46070 (local address 127.23.115.3:44857)
0620 18:38:49.796051 (+ 2554us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:49.796060 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:49.796097 (+ 37us) server_negotiation.cc:407] Connection header received
0620 18:38:49.796209 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:49.796218 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:49.797957 (+ 1739us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:49.798100 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:38:49.799798 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:49.801366 (+ 1568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:49.804320 (+ 2954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:49.806048 (+ 1728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:49.806194 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:49.806393 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:49.806623 (+ 230us) server_negotiation.cc:299] Negotiation successful
0620 18:38:49.806782 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2367,"thread_start_us":2181,"threads_started":1}
I20260620 18:38:49.807355 31199 tablet_service.cc:1467] Processing CreateTablet for tablet 0e17f2633399478296a5c3d0402d6a0f (DEFAULT_TABLE table=test-table [id=cc97f8f1c42847eba58342fa40df8db1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:38:49.808744 31199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e17f2633399478296a5c3d0402d6a0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:49.810832 31340 tablet_service.cc:1467] Processing CreateTablet for tablet 0e17f2633399478296a5c3d0402d6a0f (DEFAULT_TABLE table=test-table [id=cc97f8f1c42847eba58342fa40df8db1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:38:49.812258 31340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e17f2633399478296a5c3d0402d6a0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:49.814111 31432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:49.793464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.1:39357 (local address 127.0.0.1:43616)
0620 18:38:49.796289 (+ 2825us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:49.796304 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:49.796429 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:49.797727 (+ 1298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:49.797732 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:49.797755 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:38:49.798084 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:49.798094 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:49.799548 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:49.799552 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:49.804333 (+ 4781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:49.804342 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:49.813239 (+ 8897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:49.813247 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:49.813444 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:49.813482 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 18:38:49.813579 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 18:38:49.813730 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2704,"mutex_wait_us":24,"thread_start_us":1506,"threads_started":1}
I20260620 18:38:49.814513 31435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:49.793995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43616 (local address 127.23.115.1:39357)
0620 18:38:49.795455 (+ 1460us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:49.795466 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:49.796387 (+ 921us) server_negotiation.cc:407] Connection header received
0620 18:38:49.797437 (+ 1050us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:49.797443 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:49.797602 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:49.797731 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:38:49.798264 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:49.799412 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:49.811495 (+ 12083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:49.813005 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:49.813160 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:49.813359 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:49.813999 (+ 640us) server_negotiation.cc:299] Negotiation successful
0620 18:38:49.814167 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1272,"thread_start_us":1254,"threads_started":1}
I20260620 18:38:49.818495 31051 tablet_service.cc:1467] Processing CreateTablet for tablet 0e17f2633399478296a5c3d0402d6a0f (DEFAULT_TABLE table=test-table [id=cc97f8f1c42847eba58342fa40df8db1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:38:49.820335 31051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e17f2633399478296a5c3d0402d6a0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:49.827658 31436 tablet_bootstrap.cc:492] T 0e17f2633399478296a5c3d0402d6a0f P 7b659e72cc154daf8c7b6de5cc3d0c29: Bootstrap starting.
I20260620 18:38:49.829981 31437 tablet_bootstrap.cc:492] T 0e17f2633399478296a5c3d0402d6a0f P b8959fed0d23477fab84fd0a790447ce: Bootstrap starting.
I20260620 18:38:49.831292 31436 tablet_bootstrap.cc:654] T 0e17f2633399478296a5c3d0402d6a0f P 7b659e72cc154daf8c7b6de5cc3d0c29: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:49.832788 31436 log.cc:824] T 0e17f2633399478296a5c3d0402d6a0f P 7b659e72cc154daf8c7b6de5cc3d0c29: Log is configured to *not* fsync() on all Append() calls
I20260620 18:38:49.833455 31437 tablet_bootstrap.cc:654] T 0e17f2633399478296a5c3d0402d6a0f P b8959fed0d23477fab84fd0a790447ce: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:49.834592 31437 log.cc:824] T 0e17f2633399478296a5c3d0402d6a0f P b8959fed0d23477fab84fd0a790447ce: Log is configured to *not* fsync() on all Append() calls
I20260620 18:38:49.835564 31436 tablet_bootstrap.cc:492] T 0e17f2633399478296a5c3d0402d6a0f P 7b659e72cc154daf8c7b6de5cc3d0c29: No bootstrap required, opened a new log
I20260620 18:38:49.835843 31436 ts_tablet_manager.cc:1397] T 0e17f2633399478296a5c3d0402d6a0f P 7b659e72cc154daf8c7b6de5cc3d0c29: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.006s
I20260620 18:38:49.836711 31440 tablet_bootstrap.cc:492] T 0e17f2633399478296a5c3d0402d6a0f P 74598bd1acf5422181769e91580ab45a: Bootstrap starting.
I20260620 18:38:49.837790 31437 tablet_bootstrap.cc:492] T 0e17f2633399478296a5c3d0402d6a0f P b8959fed0d23477fab84fd0a790447ce: No bootstrap required, opened a new log
I20260620 18:38:49.838042 31437 ts_tablet_manager.cc:1397] T 0e17f2633399478296a5c3d0402d6a0f P b8959fed0d23477fab84fd0a790447ce: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260620 18:38:49.840499 31440 tablet_bootstrap.cc:654] T 0e17f2633399478296a5c3d0402d6a0f P 74598bd1acf5422181769e91580ab45a: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:49.842584 31440 log.cc:824] T 0e17f2633399478296a5c3d0402d6a0f P 74598bd1acf5422181769e91580ab45a: Log is configured to *not* fsync() on all Append() calls
I20260620 18:38:49.845891 31440 tablet_bootstrap.cc:492] T 0e17f2633399478296a5c3d0402d6a0f P 74598bd1acf5422181769e91580ab45a: No bootstrap required, opened a new log
I20260620 18:38:49.846185 31440 ts_tablet_manager.cc:1397] T 0e17f2633399478296a5c3d0402d6a0f P 74598bd1acf5422181769e91580ab45a: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260620 18:38:49.845902 31436 raft_consensus.cc:348] T 0e17f2633399478296a5c3d0402d6a0f P 7b659e72cc154daf8c7b6de5cc3d0c29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b659e72cc154daf8c7b6de5cc3d0c29" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 33167 } } peers { permanent_uuid: "b8959fed0d23477fab84fd0a790447ce" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 44857 } } peers { permanent_uuid: "74598bd1acf5422181769e91580ab45a" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 39357 } }
I20260620 18:38:49.846359 31436 raft_consensus.cc:374] T 0e17f2633399478296a5c3d0402d6a0f P 7b659e72cc154daf8c7b6de5cc3d0c29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:49.846078 31437 raft_consensus.cc:348] T 0e17f2633399478296a5c3d0402d6a0f P b8959fed0d23477fab84fd0a790447ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b659e72cc154daf8c7b6de5cc3d0c29" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 33167 } } peers { permanent_uuid: "b8959fed0d23477fab84fd0a790447ce" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 44857 } } peers { permanent_uuid: "74598bd1acf5422181769e91580ab45a" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 39357 } }
I20260620 18:38:49.846453 31436 raft_consensus.cc:729] T 0e17f2633399478296a5c3d0402d6a0f P 7b659e72cc154daf8c7b6de5cc3d0c29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b659e72cc154daf8c7b6de5cc3d0c29, State: Initialized, Role: FOLLOWER
I20260620 18:38:49.846496 31437 raft_consensus.cc:374] T 0e17f2633399478296a5c3d0402d6a0f P b8959fed0d23477fab84fd0a790447ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:49.846611 31437 raft_consensus.cc:729] T 0e17f2633399478296a5c3d0402d6a0f P b8959fed0d23477fab84fd0a790447ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8959fed0d23477fab84fd0a790447ce, State: Initialized, Role: FOLLOWER
I20260620 18:38:49.847266 31437 consensus_queue.cc:260] T 0e17f2633399478296a5c3d0402d6a0f P b8959fed0d23477fab84fd0a790447ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b659e72cc154daf8c7b6de5cc3d0c29" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 33167 } } peers { permanent_uuid: "b8959fed0d23477fab84fd0a790447ce" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 44857 } } peers { permanent_uuid: "74598bd1acf5422181769e91580ab45a" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 39357 } }
I20260620 18:38:49.847322 31436 consensus_queue.cc:260] T 0e17f2633399478296a5c3d0402d6a0f P 7b659e72cc154daf8c7b6de5cc3d0c29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b659e72cc154daf8c7b6de5cc3d0c29" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 33167 } } peers { permanent_uuid: "b8959fed0d23477fab84fd0a790447ce" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 44857 } } peers { permanent_uuid: "74598bd1acf5422181769e91580ab45a" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 39357 } }
I20260620 18:38:49.849084 31436 ts_tablet_manager.cc:1428] T 0e17f2633399478296a5c3d0402d6a0f P 7b659e72cc154daf8c7b6de5cc3d0c29: Time spent starting tablet: real 0.013s user 0.011s sys 0.003s
I20260620 18:38:49.849557 31265 heartbeater.cc:503] Master 127.23.115.62:36693 was elected leader, sending a full tablet report...
I20260620 18:38:49.853304 31406 heartbeater.cc:503] Master 127.23.115.62:36693 was elected leader, sending a full tablet report...
I20260620 18:38:49.856799 31437 ts_tablet_manager.cc:1428] T 0e17f2633399478296a5c3d0402d6a0f P b8959fed0d23477fab84fd0a790447ce: Time spent starting tablet: real 0.019s user 0.013s sys 0.000s
I20260620 18:38:49.859294 31440 raft_consensus.cc:348] T 0e17f2633399478296a5c3d0402d6a0f P 74598bd1acf5422181769e91580ab45a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b659e72cc154daf8c7b6de5cc3d0c29" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 33167 } } peers { permanent_uuid: "b8959fed0d23477fab84fd0a790447ce" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 44857 } } peers { permanent_uuid: "74598bd1acf5422181769e91580ab45a" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 39357 } }
I20260620 18:38:49.859644 31440 raft_consensus.cc:374] T 0e17f2633399478296a5c3d0402d6a0f P 74598bd1acf5422181769e91580ab45a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:49.859714 31440 raft_consensus.cc:729] T 0e17f2633399478296a5c3d0402d6a0f P 74598bd1acf5422181769e91580ab45a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74598bd1acf5422181769e91580ab45a, State: Initialized, Role: FOLLOWER
I20260620 18:38:49.860201 31440 consensus_queue.cc:260] T 0e17f2633399478296a5c3d0402d6a0f P 74598bd1acf5422181769e91580ab45a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b659e72cc154daf8c7b6de5cc3d0c29" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 33167 } } peers { permanent_uuid: "b8959fed0d23477fab84fd0a790447ce" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 44857 } } peers { permanent_uuid: "74598bd1acf5422181769e91580ab45a" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 39357 } }
I20260620 18:38:49.861691 31440 ts_tablet_manager.cc:1428] T 0e17f2633399478296a5c3d0402d6a0f P 74598bd1acf5422181769e91580ab45a: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
W20260620 18:38:49.877403 31267 tablet.cc:2369] T 0e17f2633399478296a5c3d0402d6a0f P 7b659e72cc154daf8c7b6de5cc3d0c29: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:38:49.881685 31444 raft_consensus.cc:482] T 0e17f2633399478296a5c3d0402d6a0f P 74598bd1acf5422181769e91580ab45a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:38:49.881966 31444 raft_consensus.cc:504] T 0e17f2633399478296a5c3d0402d6a0f P 74598bd1acf5422181769e91580ab45a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b659e72cc154daf8c7b6de5cc3d0c29" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 33167 } } peers { permanent_uuid: "b8959fed0d23477fab84fd0a790447ce" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 44857 } } peers { permanent_uuid: "74598bd1acf5422181769e91580ab45a" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 39357 } }
I20260620 18:38:49.883994 31444 leader_election.cc:288] T 0e17f2633399478296a5c3d0402d6a0f P 74598bd1acf5422181769e91580ab45a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b659e72cc154daf8c7b6de5cc3d0c29 (127.23.115.2:33167), b8959fed0d23477fab84fd0a790447ce (127.23.115.3:44857)
I20260620 18:38:49.890708 31434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:49.884488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.1:41247 (local address 127.23.115.3:44857)
0620 18:38:49.884694 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:49.884699 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:49.885334 (+ 635us) server_negotiation.cc:407] Connection header received
0620 18:38:49.885502 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:49.885506 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:49.885589 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:49.885692 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:38:49.886218 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:49.887034 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:49.888869 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:49.889745 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:49.889878 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:49.889935 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:49.890301 (+ 366us) server_negotiation.cc:299] Negotiation successful
0620 18:38:49.890435 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 18:38:49.890887 31431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:49.884319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.1:42657 (local address 127.23.115.2:33167)
0620 18:38:49.884547 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:49.884555 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:49.885216 (+ 661us) server_negotiation.cc:407] Connection header received
0620 18:38:49.885392 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:49.885399 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:49.885474 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:49.885538 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:38:49.886169 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:49.886884 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:49.889097 (+ 2213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:49.889909 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:49.890060 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:49.890127 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:49.890451 (+ 324us) server_negotiation.cc:299] Negotiation successful
0620 18:38:49.890596 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 18:38:49.890749 31445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:49.884293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.3:44857 (local address 127.23.115.1:41247)
0620 18:38:49.885184 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:49.885234 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:49.885362 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:49.885720 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:49.885723 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:49.885744 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:38:49.886056 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:49.886065 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:49.887167 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:49.887171 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:49.888707 (+ 1536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:49.888719 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:49.889954 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:49.889960 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:49.890127 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:49.890159 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 18:38:49.890250 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 18:38:49.890345 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":761,"mutex_wait_us":25,"thread_start_us":421,"threads_started":1}
I20260620 18:38:49.891359 31446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:49.884207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.2:33167 (local address 127.23.115.1:42657)
0620 18:38:49.885103 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:49.885120 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:49.885254 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:49.885583 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:49.885588 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:49.885614 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:38:49.886011 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:49.886028 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:49.887026 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:49.887031 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:49.888950 (+ 1919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:49.888965 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:49.890132 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:49.890137 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:49.890276 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:49.890316 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 18:38:49.890415 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 18:38:49.890554 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":782,"mutex_wait_us":78,"thread_start_us":699,"threads_started":1}
I20260620 18:38:49.892195 31360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e17f2633399478296a5c3d0402d6a0f" candidate_uuid: "74598bd1acf5422181769e91580ab45a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8959fed0d23477fab84fd0a790447ce" is_pre_election: true
I20260620 18:38:49.892280 31219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e17f2633399478296a5c3d0402d6a0f" candidate_uuid: "74598bd1acf5422181769e91580ab45a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b659e72cc154daf8c7b6de5cc3d0c29" is_pre_election: true
I20260620 18:38:49.892731 31360 raft_consensus.cc:2454] T 0e17f2633399478296a5c3d0402d6a0f P b8959fed0d23477fab84fd0a790447ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74598bd1acf5422181769e91580ab45a in term 0.
I20260620 18:38:49.892746 31219 raft_consensus.cc:2454] T 0e17f2633399478296a5c3d0402d6a0f P 7b659e72cc154daf8c7b6de5cc3d0c29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74598bd1acf5422181769e91580ab45a in term 0.
I20260620 18:38:49.893348 31005 leader_election.cc:302] T 0e17f2633399478296a5c3d0402d6a0f P 74598bd1acf5422181769e91580ab45a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74598bd1acf5422181769e91580ab45a, b8959fed0d23477fab84fd0a790447ce; no voters:
I20260620 18:38:49.893718 31444 raft_consensus.cc:2781] T 0e17f2633399478296a5c3d0402d6a0f P 74598bd1acf5422181769e91580ab45a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:38:49.893905 31444 raft_consensus.cc:482] T 0e17f2633399478296a5c3d0402d6a0f P 74598bd1acf5422181769e91580ab45a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:38:49.894014 31444 raft_consensus.cc:3037] T 0e17f2633399478296a5c3d0402d6a0f P 74598bd1acf5422181769e91580ab45a [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:49.896296 31444 raft_consensus.cc:504] T 0e17f2633399478296a5c3d0402d6a0f P 74598bd1acf5422181769e91580ab45a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b659e72cc154daf8c7b6de5cc3d0c29" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 33167 } } peers { permanent_uuid: "b8959fed0d23477fab84fd0a790447ce" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 44857 } } peers { permanent_uuid: "74598bd1acf5422181769e91580ab45a" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 39357 } }
I20260620 18:38:49.896982 31444 leader_election.cc:288] T 0e17f2633399478296a5c3d0402d6a0f P 74598bd1acf5422181769e91580ab45a [CANDIDATE]: Term 1 election: Requested vote from peers 7b659e72cc154daf8c7b6de5cc3d0c29 (127.23.115.2:33167), b8959fed0d23477fab84fd0a790447ce (127.23.115.3:44857)
I20260620 18:38:49.897455 31219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e17f2633399478296a5c3d0402d6a0f" candidate_uuid: "74598bd1acf5422181769e91580ab45a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b659e72cc154daf8c7b6de5cc3d0c29"
I20260620 18:38:49.897544 31360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e17f2633399478296a5c3d0402d6a0f" candidate_uuid: "74598bd1acf5422181769e91580ab45a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8959fed0d23477fab84fd0a790447ce"
I20260620 18:38:49.897687 31219 raft_consensus.cc:3037] T 0e17f2633399478296a5c3d0402d6a0f P 7b659e72cc154daf8c7b6de5cc3d0c29 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:49.897776 31360 raft_consensus.cc:3037] T 0e17f2633399478296a5c3d0402d6a0f P b8959fed0d23477fab84fd0a790447ce [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:49.900934 31219 raft_consensus.cc:2454] T 0e17f2633399478296a5c3d0402d6a0f P 7b659e72cc154daf8c7b6de5cc3d0c29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74598bd1acf5422181769e91580ab45a in term 1.
I20260620 18:38:49.900934 31360 raft_consensus.cc:2454] T 0e17f2633399478296a5c3d0402d6a0f P b8959fed0d23477fab84fd0a790447ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74598bd1acf5422181769e91580ab45a in term 1.
I20260620 18:38:49.901400 31005 leader_election.cc:302] T 0e17f2633399478296a5c3d0402d6a0f P 74598bd1acf5422181769e91580ab45a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74598bd1acf5422181769e91580ab45a, b8959fed0d23477fab84fd0a790447ce; no voters:
I20260620 18:38:49.901682 31444 raft_consensus.cc:2781] T 0e17f2633399478296a5c3d0402d6a0f P 74598bd1acf5422181769e91580ab45a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:38:49.902281 31444 raft_consensus.cc:686] T 0e17f2633399478296a5c3d0402d6a0f P 74598bd1acf5422181769e91580ab45a [term 1 LEADER]: Becoming Leader. State: Replica: 74598bd1acf5422181769e91580ab45a, State: Running, Role: LEADER
I20260620 18:38:49.902869 31444 consensus_queue.cc:237] T 0e17f2633399478296a5c3d0402d6a0f P 74598bd1acf5422181769e91580ab45a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b659e72cc154daf8c7b6de5cc3d0c29" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 33167 } } peers { permanent_uuid: "b8959fed0d23477fab84fd0a790447ce" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 44857 } } peers { permanent_uuid: "74598bd1acf5422181769e91580ab45a" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 39357 } }
I20260620 18:38:49.908124 30148 catalog_manager.cc:5495] T 0e17f2633399478296a5c3d0402d6a0f P 74598bd1acf5422181769e91580ab45a reported cstate change: term changed from 0 to 1, leader changed from <none> to 74598bd1acf5422181769e91580ab45a (127.23.115.1). New cstate: current_term: 1 leader_uuid: "74598bd1acf5422181769e91580ab45a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b659e72cc154daf8c7b6de5cc3d0c29" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 33167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8959fed0d23477fab84fd0a790447ce" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 44857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74598bd1acf5422181769e91580ab45a" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 39357 } health_report { overall_health: HEALTHY } } }
I20260620 18:38:49.941712 31435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:49.936161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43632 (local address 127.23.115.1:39357)
0620 18:38:49.936320 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:49.936324 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:49.936342 (+ 18us) server_negotiation.cc:407] Connection header received
0620 18:38:49.936430 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:49.936434 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:49.936509 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:49.936636 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:38:49.937036 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:49.937782 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:49.939473 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:49.939921 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:49.940024 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:49.940549 (+ 525us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:38:49.941041 (+ 492us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:38:49.941174 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:49.941418 (+ 244us) server_negotiation.cc:299] Negotiation successful
0620 18:38:49.941490 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 18:38:49.949836 31360 raft_consensus.cc:1261] T 0e17f2633399478296a5c3d0402d6a0f P b8959fed0d23477fab84fd0a790447ce [term 1 FOLLOWER]: Refusing update from remote peer 74598bd1acf5422181769e91580ab45a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:38:49.949838 31219 raft_consensus.cc:1261] T 0e17f2633399478296a5c3d0402d6a0f P 7b659e72cc154daf8c7b6de5cc3d0c29 [term 1 FOLLOWER]: Refusing update from remote peer 74598bd1acf5422181769e91580ab45a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:38:49.950958 31447 consensus_queue.cc:1035] T 0e17f2633399478296a5c3d0402d6a0f P 74598bd1acf5422181769e91580ab45a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8959fed0d23477fab84fd0a790447ce" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 44857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:38:49.951332 31444 consensus_queue.cc:1035] T 0e17f2633399478296a5c3d0402d6a0f P 74598bd1acf5422181769e91580ab45a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b659e72cc154daf8c7b6de5cc3d0c29" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 33167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 18:38:49.955909 31408 tablet.cc:2369] T 0e17f2633399478296a5c3d0402d6a0f P b8959fed0d23477fab84fd0a790447ce: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:38:49.968413 31450 mvcc.cc:205] Tried to move back new op lower bound from 7298993069864968192 to 7298993069686525952. Current Snapshot: MvccSnapshot[applied={T|T < 7298993069864968192}]
I20260620 18:38:49.970197 31452 mvcc.cc:205] Tried to move back new op lower bound from 7298993069864968192 to 7298993069686525952. Current Snapshot: MvccSnapshot[applied={T|T < 7298993069864968192}]
I20260620 18:38:49.986380 30148 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47002:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:38:49.986630 30148 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47002:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:38:49.993072 30148 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8279b148b2454bfb90afa089240c8086: Sending DeleteTablet for 3 replicas of tablet 0e17f2633399478296a5c3d0402d6a0f
I20260620 18:38:49.995079 31340 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e17f2633399478296a5c3d0402d6a0f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:38:49 UTC) from {username='oryx', principal='oryx/127.23.115.62@KRBTEST.COM'} at 127.0.0.1:46070
I20260620 18:38:49.995090 31199 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e17f2633399478296a5c3d0402d6a0f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:38:49 UTC) from {username='oryx', principal='oryx/127.23.115.62@KRBTEST.COM'} at 127.0.0.1:38116
I20260620 18:38:49.995886 31051 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e17f2633399478296a5c3d0402d6a0f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:38:49 UTC) from {username='oryx', principal='oryx/127.23.115.62@KRBTEST.COM'} at 127.0.0.1:43616
I20260620 18:38:50.006367 31461 tablet_replica.cc:330] T 0e17f2633399478296a5c3d0402d6a0f P 74598bd1acf5422181769e91580ab45a: stopping tablet replica
I20260620 18:38:50.007117 31461 raft_consensus.cc:2229] T 0e17f2633399478296a5c3d0402d6a0f P 74598bd1acf5422181769e91580ab45a [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:38:50.007934 31461 raft_consensus.cc:2258] T 0e17f2633399478296a5c3d0402d6a0f P 74598bd1acf5422181769e91580ab45a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:38:50.011579 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 30117
I20260620 18:38:50.011876 31460 tablet_replica.cc:330] T 0e17f2633399478296a5c3d0402d6a0f P 7b659e72cc154daf8c7b6de5cc3d0c29: stopping tablet replica
I20260620 18:38:50.013031 30179 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:38:50.013711 31460 raft_consensus.cc:2229] T 0e17f2633399478296a5c3d0402d6a0f P 7b659e72cc154daf8c7b6de5cc3d0c29 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:38:50.017495 31460 raft_consensus.cc:2258] T 0e17f2633399478296a5c3d0402d6a0f P 7b659e72cc154daf8c7b6de5cc3d0c29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:38:50.019207 31461 ts_tablet_manager.cc:1905] T 0e17f2633399478296a5c3d0402d6a0f P 74598bd1acf5422181769e91580ab45a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:38:50.022188 31460 ts_tablet_manager.cc:1905] T 0e17f2633399478296a5c3d0402d6a0f P 7b659e72cc154daf8c7b6de5cc3d0c29: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:38:50.023663 31459 tablet_replica.cc:330] T 0e17f2633399478296a5c3d0402d6a0f P b8959fed0d23477fab84fd0a790447ce: stopping tablet replica
I20260620 18:38:50.024124 31459 raft_consensus.cc:2229] T 0e17f2633399478296a5c3d0402d6a0f P b8959fed0d23477fab84fd0a790447ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:38:50.024652 31459 raft_consensus.cc:2258] T 0e17f2633399478296a5c3d0402d6a0f P b8959fed0d23477fab84fd0a790447ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:38:50.029726 31460 ts_tablet_manager.cc:1918] T 0e17f2633399478296a5c3d0402d6a0f P 7b659e72cc154daf8c7b6de5cc3d0c29: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:38:50.030030 31460 log.cc:1196] T 0e17f2633399478296a5c3d0402d6a0f P 7b659e72cc154daf8c7b6de5cc3d0c29: Deleting WAL directory at /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/ts-1/wal/wals/0e17f2633399478296a5c3d0402d6a0f
I20260620 18:38:50.030671 31460 ts_tablet_manager.cc:1939] T 0e17f2633399478296a5c3d0402d6a0f P 7b659e72cc154daf8c7b6de5cc3d0c29: Deleting consensus metadata
I20260620 18:38:50.036077 31461 ts_tablet_manager.cc:1918] T 0e17f2633399478296a5c3d0402d6a0f P 74598bd1acf5422181769e91580ab45a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:38:50.036266 31461 log.cc:1196] T 0e17f2633399478296a5c3d0402d6a0f P 74598bd1acf5422181769e91580ab45a: Deleting WAL directory at /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/ts-0/wal/wals/0e17f2633399478296a5c3d0402d6a0f
I20260620 18:38:50.036842 31461 ts_tablet_manager.cc:1939] T 0e17f2633399478296a5c3d0402d6a0f P 74598bd1acf5422181769e91580ab45a: Deleting consensus metadata
I20260620 18:38:50.038136 31459 ts_tablet_manager.cc:1905] T 0e17f2633399478296a5c3d0402d6a0f P b8959fed0d23477fab84fd0a790447ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:38:50.045979 31459 ts_tablet_manager.cc:1918] T 0e17f2633399478296a5c3d0402d6a0f P b8959fed0d23477fab84fd0a790447ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:38:50.046164 31459 log.cc:1196] T 0e17f2633399478296a5c3d0402d6a0f P b8959fed0d23477fab84fd0a790447ce: Deleting WAL directory at /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/ts-2/wal/wals/0e17f2633399478296a5c3d0402d6a0f
I20260620 18:38:50.046686 31459 ts_tablet_manager.cc:1939] T 0e17f2633399478296a5c3d0402d6a0f P b8959fed0d23477fab84fd0a790447ce: Deleting consensus metadata
I20260620 18:38:50.049000 31413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:49.997527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34586 (local address 127.23.115.61:45651)
0620 18:38:49.997722 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:49.997729 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:50.009042 (+ 11313us) server_negotiation.cc:407] Connection header received
0620 18:38:50.009252 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:50.009257 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:50.009346 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:50.009454 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:50.017076 (+ 7622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:50.018189 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:50.036945 (+ 18756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:50.037584 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:50.037718 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:50.042269 (+ 4551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:50.042276 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:50.042281 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:50.042327 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:50.045813 (+ 3486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:50.046979 (+ 1166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:50.046985 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:50.046987 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:50.047143 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:50.047540 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:50.047544 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:50.047547 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:50.047926 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:50.048185 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:50.048536 (+ 351us) server_negotiation.cc:299] Negotiation successful
0620 18:38:50.048729 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 18:38:50.065294 31409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:50.010702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38362 (local address 127.23.115.60:33051)
0620 18:38:50.011075 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:50.011083 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:50.049497 (+ 38414us) server_negotiation.cc:407] Connection header received
0620 18:38:50.049763 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:50.049771 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:50.049862 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:50.049958 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:50.051811 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:50.052832 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:50.054613 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:50.055295 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:50.055384 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:50.058843 (+ 3459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:50.058850 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:50.058855 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:50.058903 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:50.062380 (+ 3477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:50.063221 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:50.063227 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:50.063232 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:50.063375 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:50.063792 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:50.063797 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:50.063801 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:50.064146 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:50.064402 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:50.064701 (+ 299us) server_negotiation.cc:299] Negotiation successful
0620 18:38:50.064916 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 18:38:50.168533 30134 catalog_manager.cc:4841] TS 74598bd1acf5422181769e91580ab45a (127.23.115.1:39357): tablet 0e17f2633399478296a5c3d0402d6a0f (table test-table [id=cc97f8f1c42847eba58342fa40df8db1]) successfully deleted
I20260620 18:38:50.169750 30134 catalog_manager.cc:4841] TS 7b659e72cc154daf8c7b6de5cc3d0c29 (127.23.115.2:33167): tablet 0e17f2633399478296a5c3d0402d6a0f (table test-table [id=cc97f8f1c42847eba58342fa40df8db1]) successfully deleted
I20260620 18:38:50.169778 30136 catalog_manager.cc:4841] TS b8959fed0d23477fab84fd0a790447ce (127.23.115.3:44857): tablet 0e17f2633399478296a5c3d0402d6a0f (table test-table [id=cc97f8f1c42847eba58342fa40df8db1]) successfully deleted
I20260620 18:38:50.170326 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 30117
I20260620 18:38:50.209259 31413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:50.194128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34588 (local address 127.23.115.61:45651)
0620 18:38:50.194340 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:50.194346 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:50.194366 (+ 20us) server_negotiation.cc:407] Connection header received
0620 18:38:50.194597 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:50.194603 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:50.194686 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:50.194760 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:50.196705 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:50.197975 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:50.199886 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:50.200375 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:50.200499 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:50.203238 (+ 2739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:50.203243 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:50.203246 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:50.203281 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:50.206352 (+ 3071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:50.207066 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:50.207071 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:50.207074 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:50.207222 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:50.207674 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:50.207679 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:50.207682 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:50.208058 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:50.208284 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:50.208659 (+ 375us) server_negotiation.cc:299] Negotiation successful
0620 18:38:50.208866 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 18:38:50.210429 31409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:50.195001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38378 (local address 127.23.115.60:33051)
0620 18:38:50.195240 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:50.195246 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:50.195317 (+ 71us) server_negotiation.cc:407] Connection header received
0620 18:38:50.195493 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:50.195499 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:50.195576 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:50.195684 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:50.197753 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:50.198545 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:50.199958 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:50.200614 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:50.200730 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:50.203988 (+ 3258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:50.203993 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:50.203996 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:50.204028 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:50.207197 (+ 3169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:50.207976 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:50.207982 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:50.207986 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:50.208107 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:50.208513 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:50.208518 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:50.208521 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:50.208869 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:50.209089 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:50.209723 (+ 634us) server_negotiation.cc:299] Negotiation successful
0620 18:38:50.210115 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260620 18:38:50.980513 31470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:50.979388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:36693 (local address 127.23.115.2:42185)
0620 18:38:50.979880 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:50.980030 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:36693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":279,"threads_started":1}
W20260620 18:38:50.981055 31265 heartbeater.cc:644] Failed to heartbeat to 127.23.115.62:36693 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.23.115.62:36693: connect: Connection refused (error 111)
I20260620 18:38:50.981884 31471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:50.980580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:36693 (local address 127.23.115.3:45051)
0620 18:38:50.981521 (+ 941us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:50.981662 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:36693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":817,"threads_started":1}
I20260620 18:38:50.982445 31472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:50.981280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:36693 (local address 127.23.115.1:49789)
0620 18:38:50.982110 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:50.982245 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:36693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":714,"thread_start_us":620,"threads_started":1}
W20260620 18:38:50.982707 31406 heartbeater.cc:644] Failed to heartbeat to 127.23.115.62:36693 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.23.115.62:36693: connect: Connection refused (error 111)
W20260620 18:38:50.983088 31117 heartbeater.cc:644] Failed to heartbeat to 127.23.115.62:36693 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.23.115.62:36693: connect: Connection refused (error 111)
I20260620 18:38:51.494905 31474 raft_consensus.cc:482] T 00000000000000000000000000000000 P 294609dc6da34eefbb57726f025c5619 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8279b148b2454bfb90afa089240c8086)
I20260620 18:38:51.494761 31473 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d00dab869714576ae199f6caf0c39d8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8279b148b2454bfb90afa089240c8086)
I20260620 18:38:51.495187 31474 raft_consensus.cc:504] T 00000000000000000000000000000000 P 294609dc6da34eefbb57726f025c5619 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8279b148b2454bfb90afa089240c8086" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 36693 } } peers { permanent_uuid: "0d00dab869714576ae199f6caf0c39d8" member_type: VOTER last_known_addr { host: "127.23.115.61" port: 45651 } attrs { promote: false } } peers { permanent_uuid: "294609dc6da34eefbb57726f025c5619" member_type: VOTER last_known_addr { host: "127.23.115.60" port: 33051 } attrs { promote: false } }
I20260620 18:38:51.495290 31473 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d00dab869714576ae199f6caf0c39d8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8279b148b2454bfb90afa089240c8086" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 36693 } } peers { permanent_uuid: "0d00dab869714576ae199f6caf0c39d8" member_type: VOTER last_known_addr { host: "127.23.115.61" port: 45651 } attrs { promote: false } } peers { permanent_uuid: "294609dc6da34eefbb57726f025c5619" member_type: VOTER last_known_addr { host: "127.23.115.60" port: 33051 } attrs { promote: false } }
I20260620 18:38:51.496493 31473 leader_election.cc:288] T 00000000000000000000000000000000 P 0d00dab869714576ae199f6caf0c39d8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8279b148b2454bfb90afa089240c8086 (127.23.115.62:36693), 294609dc6da34eefbb57726f025c5619 (127.23.115.60:33051)
I20260620 18:38:51.496521 31474 leader_election.cc:288] T 00000000000000000000000000000000 P 294609dc6da34eefbb57726f025c5619 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8279b148b2454bfb90afa089240c8086 (127.23.115.62:36693), 0d00dab869714576ae199f6caf0c39d8 (127.23.115.61:45651)
I20260620 18:38:51.499903 31476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:51.496682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:36693 (local address 127.0.0.1:47354)
0620 18:38:51.499405 (+ 2723us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:51.499634 (+ 229us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:36693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2568,"thread_start_us":2406,"threads_started":1}
I20260620 18:38:51.500455 31475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:51.496678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:36693 (local address 127.0.0.1:47352)
0620 18:38:51.499836 (+ 3158us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:51.500163 (+ 327us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:36693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3020,"thread_start_us":2893,"threads_started":1}
W20260620 18:38:51.500780 30921 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.115.62:36693: connect: Connection refused (error 111)
W20260620 18:38:51.502059 30733 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.115.62:36693: connect: Connection refused (error 111)
I20260620 18:38:51.503052 31477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:51.502268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:36693 (local address 127.0.0.1:47362)
0620 18:38:51.502458 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:51.502697 (+ 239us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:36693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260620 18:38:51.503307 30921 leader_election.cc:334] T 00000000000000000000000000000000 P 294609dc6da34eefbb57726f025c5619 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8279b148b2454bfb90afa089240c8086 (127.23.115.62:36693): Network error: Client connection negotiation failed: client connection to 127.23.115.62:36693: connect: Connection refused (error 111)
I20260620 18:38:51.505216 31478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:51.504525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:36693 (local address 127.0.0.1:47370)
0620 18:38:51.504710 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:51.504938 (+ 228us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:36693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260620 18:38:51.505407 30733 leader_election.cc:334] T 00000000000000000000000000000000 P 0d00dab869714576ae199f6caf0c39d8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8279b148b2454bfb90afa089240c8086 (127.23.115.62:36693): Network error: Client connection negotiation failed: client connection to 127.23.115.62:36693: connect: Connection refused (error 111)
I20260620 18:38:51.509126 31479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:51.500738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38390 (local address 127.23.115.60:33051)
0620 18:38:51.501393 (+ 655us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:51.501399 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:51.501418 (+ 19us) server_negotiation.cc:407] Connection header received
0620 18:38:51.501492 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:51.501498 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:51.501576 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:51.502239 (+ 663us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:38:51.502445 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:51.503777 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:51.506207 (+ 2430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:51.508013 (+ 1806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:51.508105 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:51.508172 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:51.508723 (+ 551us) server_negotiation.cc:299] Negotiation successful
0620 18:38:51.508840 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":531,"thread_start_us":375,"threads_started":1}
I20260620 18:38:51.509346 31475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:51.500166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.60:33051 (local address 127.0.0.1:38390)
0620 18:38:51.500716 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:51.500733 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:51.500853 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:51.501734 (+ 881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:51.501739 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:51.501772 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:38:51.502091 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:51.502104 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:51.503935 (+ 1831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:51.503941 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:51.506021 (+ 2080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:51.506035 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:51.508162 (+ 2127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:51.508169 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:51.508344 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:51.508381 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 18:38:51.508718 (+ 337us) client_negotiation.cc:240] Negotiation successful
0620 18:38:51.509007 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"mutex_wait_us":3,"thread_start_us":1129,"threads_started":1}
I20260620 18:38:51.509909 31476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:51.499477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.61:45651 (local address 127.0.0.1:34600)
0620 18:38:51.500103 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:51.500120 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:51.500249 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:51.502707 (+ 2458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:51.502712 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:51.502736 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:38:51.503126 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:51.503138 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:51.505431 (+ 2293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:51.505437 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:51.507418 (+ 1981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:51.507433 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:51.509135 (+ 1702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:51.509142 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:51.509318 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:51.509352 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 18:38:51.509453 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 18:38:51.509586 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"mutex_wait_us":34,"thread_start_us":993,"threads_started":1}
I20260620 18:38:51.509907 31481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:51.501511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34600 (local address 127.23.115.61:45651)
0620 18:38:51.502409 (+ 898us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:51.502414 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:51.502429 (+ 15us) server_negotiation.cc:407] Connection header received
0620 18:38:51.502475 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:51.502479 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:51.502540 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:51.503346 (+ 806us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:38:51.503685 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:51.505163 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:51.507594 (+ 2431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:51.508829 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:51.509061 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:51.509475 (+ 414us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:51.509531 (+ 56us) server_negotiation.cc:299] Negotiation successful
0620 18:38:51.509674 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":51,"server-negotiator.queue_time_us":474,"thread_start_us":291,"threads_started":1}
I20260620 18:38:51.510087 30947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0d00dab869714576ae199f6caf0c39d8" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "294609dc6da34eefbb57726f025c5619" is_pre_election: true
I20260620 18:38:51.510423 30947 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 294609dc6da34eefbb57726f025c5619 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d00dab869714576ae199f6caf0c39d8 in term 1.
I20260620 18:38:51.510613 30758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "294609dc6da34eefbb57726f025c5619" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0d00dab869714576ae199f6caf0c39d8" is_pre_election: true
I20260620 18:38:51.510896 30758 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0d00dab869714576ae199f6caf0c39d8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 294609dc6da34eefbb57726f025c5619 in term 1.
I20260620 18:38:51.510996 30733 leader_election.cc:302] T 00000000000000000000000000000000 P 0d00dab869714576ae199f6caf0c39d8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d00dab869714576ae199f6caf0c39d8, 294609dc6da34eefbb57726f025c5619; no voters: 8279b148b2454bfb90afa089240c8086
I20260620 18:38:51.511391 31473 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d00dab869714576ae199f6caf0c39d8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 18:38:51.511973 30921 leader_election.cc:302] T 00000000000000000000000000000000 P 294609dc6da34eefbb57726f025c5619 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d00dab869714576ae199f6caf0c39d8, 294609dc6da34eefbb57726f025c5619; no voters: 8279b148b2454bfb90afa089240c8086
I20260620 18:38:51.512326 31474 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 294609dc6da34eefbb57726f025c5619 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 18:38:51.512450 31474 raft_consensus.cc:482] T 00000000000000000000000000000000 P 294609dc6da34eefbb57726f025c5619 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8279b148b2454bfb90afa089240c8086)
I20260620 18:38:51.512537 31474 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 294609dc6da34eefbb57726f025c5619 [term 1 FOLLOWER]: Advancing to term 2
I20260620 18:38:51.512588 31473 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d00dab869714576ae199f6caf0c39d8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8279b148b2454bfb90afa089240c8086)
I20260620 18:38:51.512706 31473 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d00dab869714576ae199f6caf0c39d8 [term 1 FOLLOWER]: Advancing to term 2
I20260620 18:38:51.514592 31474 raft_consensus.cc:504] T 00000000000000000000000000000000 P 294609dc6da34eefbb57726f025c5619 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8279b148b2454bfb90afa089240c8086" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 36693 } } peers { permanent_uuid: "0d00dab869714576ae199f6caf0c39d8" member_type: VOTER last_known_addr { host: "127.23.115.61" port: 45651 } attrs { promote: false } } peers { permanent_uuid: "294609dc6da34eefbb57726f025c5619" member_type: VOTER last_known_addr { host: "127.23.115.60" port: 33051 } attrs { promote: false } }
I20260620 18:38:51.514761 31473 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d00dab869714576ae199f6caf0c39d8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8279b148b2454bfb90afa089240c8086" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 36693 } } peers { permanent_uuid: "0d00dab869714576ae199f6caf0c39d8" member_type: VOTER last_known_addr { host: "127.23.115.61" port: 45651 } attrs { promote: false } } peers { permanent_uuid: "294609dc6da34eefbb57726f025c5619" member_type: VOTER last_known_addr { host: "127.23.115.60" port: 33051 } attrs { promote: false } }
I20260620 18:38:51.515244 31474 leader_election.cc:288] T 00000000000000000000000000000000 P 294609dc6da34eefbb57726f025c5619 [CANDIDATE]: Term 2 election: Requested vote from peers 8279b148b2454bfb90afa089240c8086 (127.23.115.62:36693), 0d00dab869714576ae199f6caf0c39d8 (127.23.115.61:45651)
I20260620 18:38:51.515410 31473 leader_election.cc:288] T 00000000000000000000000000000000 P 0d00dab869714576ae199f6caf0c39d8 [CANDIDATE]: Term 2 election: Requested vote from peers 8279b148b2454bfb90afa089240c8086 (127.23.115.62:36693), 294609dc6da34eefbb57726f025c5619 (127.23.115.60:33051)
I20260620 18:38:51.515817 31476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:51.515416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:36693 (local address 127.0.0.1:47398)
0620 18:38:51.515562 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:51.515649 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:36693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 18:38:51.515911 31475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:51.515547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:36693 (local address 127.0.0.1:47408)
0620 18:38:51.515676 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:51.515759 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:36693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260620 18:38:51.516057 30758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "294609dc6da34eefbb57726f025c5619" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0d00dab869714576ae199f6caf0c39d8"
I20260620 18:38:51.516281 30758 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 0d00dab869714576ae199f6caf0c39d8 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 294609dc6da34eefbb57726f025c5619 in current term 2: Already voted for candidate 0d00dab869714576ae199f6caf0c39d8 in this term.
I20260620 18:38:51.516552 30947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0d00dab869714576ae199f6caf0c39d8" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "294609dc6da34eefbb57726f025c5619"
I20260620 18:38:51.516768 31475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:51.516420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:36693 (local address 127.0.0.1:47412)
0620 18:38:51.516554 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:51.516628 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:36693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260620 18:38:51.516765 30947 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 294609dc6da34eefbb57726f025c5619 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d00dab869714576ae199f6caf0c39d8 in current term 2: Already voted for candidate 294609dc6da34eefbb57726f025c5619 in this term.
W20260620 18:38:51.517060 30733 leader_election.cc:334] T 00000000000000000000000000000000 P 0d00dab869714576ae199f6caf0c39d8 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 8279b148b2454bfb90afa089240c8086 (127.23.115.62:36693): Network error: Client connection negotiation failed: client connection to 127.23.115.62:36693: connect: Connection refused (error 111)
I20260620 18:38:51.517370 30733 leader_election.cc:302] T 00000000000000000000000000000000 P 0d00dab869714576ae199f6caf0c39d8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d00dab869714576ae199f6caf0c39d8; no voters: 294609dc6da34eefbb57726f025c5619, 8279b148b2454bfb90afa089240c8086
I20260620 18:38:51.517601 31473 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 0d00dab869714576ae199f6caf0c39d8 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260620 18:38:51.517724 31476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:51.517402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:36693 (local address 127.0.0.1:47420)
0620 18:38:51.517549 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:51.517612 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:36693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260620 18:38:51.517865 30921 leader_election.cc:334] T 00000000000000000000000000000000 P 294609dc6da34eefbb57726f025c5619 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 8279b148b2454bfb90afa089240c8086 (127.23.115.62:36693): Network error: Client connection negotiation failed: client connection to 127.23.115.62:36693: connect: Connection refused (error 111)
I20260620 18:38:51.517988 30921 leader_election.cc:302] T 00000000000000000000000000000000 P 294609dc6da34eefbb57726f025c5619 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 294609dc6da34eefbb57726f025c5619; no voters: 0d00dab869714576ae199f6caf0c39d8, 8279b148b2454bfb90afa089240c8086
I20260620 18:38:51.518339 31474 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 294609dc6da34eefbb57726f025c5619 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260620 18:38:51.983419 31483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:51.981961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:36693 (local address 127.23.115.2:33203)
0620 18:38:51.982469 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:51.982586 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:36693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":243,"threads_started":1}
I20260620 18:38:51.985215 31484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:51.983764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:36693 (local address 127.23.115.1:54225)
0620 18:38:51.984915 (+ 1151us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:51.985029 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:36693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1050,"thread_start_us":961,"threads_started":1}
I20260620 18:38:51.985721 31485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:51.984270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:36693 (local address 127.23.115.3:59129)
0620 18:38:51.984967 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:51.985453 (+ 486us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:36693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":488,"threads_started":1}
I20260620 18:38:52.986253 31486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:52.985436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:36693 (local address 127.23.115.2:55851)
0620 18:38:52.985893 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:52.986005 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:36693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":247,"threads_started":1}
I20260620 18:38:52.987901 31488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:52.987070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:36693 (local address 127.23.115.3:59621)
0620 18:38:52.987521 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:52.987615 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:36693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"threads_started":1}
I20260620 18:38:52.988395 31487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:52.986612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:36693 (local address 127.23.115.1:42513)
0620 18:38:52.987206 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:52.988170 (+ 964us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:36693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":392,"threads_started":1}
I20260620 18:38:53.324419 31489 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d00dab869714576ae199f6caf0c39d8 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:38:53.324610 31489 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d00dab869714576ae199f6caf0c39d8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8279b148b2454bfb90afa089240c8086" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 36693 } } peers { permanent_uuid: "0d00dab869714576ae199f6caf0c39d8" member_type: VOTER last_known_addr { host: "127.23.115.61" port: 45651 } attrs { promote: false } } peers { permanent_uuid: "294609dc6da34eefbb57726f025c5619" member_type: VOTER last_known_addr { host: "127.23.115.60" port: 33051 } attrs { promote: false } }
I20260620 18:38:53.325268 31489 leader_election.cc:288] T 00000000000000000000000000000000 P 0d00dab869714576ae199f6caf0c39d8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8279b148b2454bfb90afa089240c8086 (127.23.115.62:36693), 294609dc6da34eefbb57726f025c5619 (127.23.115.60:33051)
I20260620 18:38:53.326305 31490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:53.325495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:36693 (local address 127.0.0.1:47668)
0620 18:38:53.325972 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:53.326103 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:36693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":236,"threads_started":1}
I20260620 18:38:53.326537 30947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0d00dab869714576ae199f6caf0c39d8" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "294609dc6da34eefbb57726f025c5619" is_pre_election: true
I20260620 18:38:53.326783 30947 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 294609dc6da34eefbb57726f025c5619 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d00dab869714576ae199f6caf0c39d8 in term 2.
I20260620 18:38:53.327571 30733 leader_election.cc:302] T 00000000000000000000000000000000 P 0d00dab869714576ae199f6caf0c39d8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d00dab869714576ae199f6caf0c39d8, 294609dc6da34eefbb57726f025c5619; no voters:
I20260620 18:38:53.327828 31489 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d00dab869714576ae199f6caf0c39d8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 18:38:53.328040 31489 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d00dab869714576ae199f6caf0c39d8 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:38:53.328115 31490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:53.327367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:36693 (local address 127.0.0.1:47684)
0620 18:38:53.327634 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:53.327796 (+ 162us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:36693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260620 18:38:53.328137 31489 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d00dab869714576ae199f6caf0c39d8 [term 2 FOLLOWER]: Advancing to term 3
W20260620 18:38:53.328388 30733 leader_election.cc:334] T 00000000000000000000000000000000 P 0d00dab869714576ae199f6caf0c39d8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8279b148b2454bfb90afa089240c8086 (127.23.115.62:36693): Network error: Client connection negotiation failed: client connection to 127.23.115.62:36693: connect: Connection refused (error 111)
I20260620 18:38:53.330294 31489 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d00dab869714576ae199f6caf0c39d8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8279b148b2454bfb90afa089240c8086" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 36693 } } peers { permanent_uuid: "0d00dab869714576ae199f6caf0c39d8" member_type: VOTER last_known_addr { host: "127.23.115.61" port: 45651 } attrs { promote: false } } peers { permanent_uuid: "294609dc6da34eefbb57726f025c5619" member_type: VOTER last_known_addr { host: "127.23.115.60" port: 33051 } attrs { promote: false } }
I20260620 18:38:53.330785 31489 leader_election.cc:288] T 00000000000000000000000000000000 P 0d00dab869714576ae199f6caf0c39d8 [CANDIDATE]: Term 3 election: Requested vote from peers 8279b148b2454bfb90afa089240c8086 (127.23.115.62:36693), 294609dc6da34eefbb57726f025c5619 (127.23.115.60:33051)
I20260620 18:38:53.331436 31490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:53.330919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:36693 (local address 127.0.0.1:47700)
0620 18:38:53.331134 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:53.331215 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:36693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260620 18:38:53.331512 30947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0d00dab869714576ae199f6caf0c39d8" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "294609dc6da34eefbb57726f025c5619"
I20260620 18:38:53.331743 30947 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 294609dc6da34eefbb57726f025c5619 [term 2 FOLLOWER]: Advancing to term 3
I20260620 18:38:53.332423 31490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:53.332070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:36693 (local address 127.0.0.1:47708)
0620 18:38:53.332208 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:53.332297 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:36693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 18:38:53.332562 30733 leader_election.cc:334] T 00000000000000000000000000000000 P 0d00dab869714576ae199f6caf0c39d8 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 8279b148b2454bfb90afa089240c8086 (127.23.115.62:36693): Network error: Client connection negotiation failed: client connection to 127.23.115.62:36693: connect: Connection refused (error 111)
I20260620 18:38:53.333925 30947 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 294609dc6da34eefbb57726f025c5619 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d00dab869714576ae199f6caf0c39d8 in term 3.
I20260620 18:38:53.334282 30733 leader_election.cc:302] T 00000000000000000000000000000000 P 0d00dab869714576ae199f6caf0c39d8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d00dab869714576ae199f6caf0c39d8, 294609dc6da34eefbb57726f025c5619; no voters: 8279b148b2454bfb90afa089240c8086
I20260620 18:38:53.334576 31489 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d00dab869714576ae199f6caf0c39d8 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 18:38:53.334975 31489 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d00dab869714576ae199f6caf0c39d8 [term 3 LEADER]: Becoming Leader. State: Replica: 0d00dab869714576ae199f6caf0c39d8, State: Running, Role: LEADER
I20260620 18:38:53.335331 31489 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d00dab869714576ae199f6caf0c39d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8279b148b2454bfb90afa089240c8086" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 36693 } } peers { permanent_uuid: "0d00dab869714576ae199f6caf0c39d8" member_type: VOTER last_known_addr { host: "127.23.115.61" port: 45651 } attrs { promote: false } } peers { permanent_uuid: "294609dc6da34eefbb57726f025c5619" member_type: VOTER last_known_addr { host: "127.23.115.60" port: 33051 } attrs { promote: false } }
I20260620 18:38:53.337391 31492 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d00dab869714576ae199f6caf0c39d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d00dab869714576ae199f6caf0c39d8. Latest consensus state: current_term: 3 leader_uuid: "0d00dab869714576ae199f6caf0c39d8" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8279b148b2454bfb90afa089240c8086" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 36693 } } peers { permanent_uuid: "0d00dab869714576ae199f6caf0c39d8" member_type: VOTER last_known_addr { host: "127.23.115.61" port: 45651 } attrs { promote: false } } peers { permanent_uuid: "294609dc6da34eefbb57726f025c5619" member_type: VOTER last_known_addr { host: "127.23.115.60" port: 33051 } attrs { promote: false } } }
I20260620 18:38:53.337636 31492 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d00dab869714576ae199f6caf0c39d8 [sys.catalog]: This master's current role is: LEADER
I20260620 18:38:53.338382 31494 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:38:53.342187 31494 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:38:53.343148 31494 catalog_manager.cc:1259] Loaded cluster ID: f0357000094b4f94b558a6c421dae89d
I20260620 18:38:53.343221 31494 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:38:53.344550 31494 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:38:53.345427 31494 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0d00dab869714576ae199f6caf0c39d8: Loaded TSK: 0
I20260620 18:38:53.346221 31494 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:38:53.430657 30746 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34588:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:38:53.437506 30746 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:38:53.441365 31490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:53.440989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:36693 (local address 127.0.0.1:47754)
0620 18:38:53.441136 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:53.441225 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:36693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260620 18:38:53.441778 30947 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 294609dc6da34eefbb57726f025c5619 [term 3 FOLLOWER]: Refusing update from remote peer 0d00dab869714576ae199f6caf0c39d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260620 18:38:53.442313 31490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:53.441965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:36693 (local address 127.0.0.1:47760)
0620 18:38:53.442088 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:53.442169 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:36693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260620 18:38:53.442461 30733 consensus_peers.cc:487] T 00000000000000000000000000000000 P 0d00dab869714576ae199f6caf0c39d8 -> Peer 8279b148b2454bfb90afa089240c8086 (127.23.115.62:36693): Couldn't send request to peer 8279b148b2454bfb90afa089240c8086. Status: Network error: Client connection negotiation failed: client connection to 127.23.115.62:36693: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 18:38:53.442487 31492 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0d00dab869714576ae199f6caf0c39d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "294609dc6da34eefbb57726f025c5619" member_type: VOTER last_known_addr { host: "127.23.115.60" port: 33051 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260620 18:38:53.445664 31496 sys_catalog.cc:455] T 00000000000000000000000000000000 P 294609dc6da34eefbb57726f025c5619 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d00dab869714576ae199f6caf0c39d8. Latest consensus state: current_term: 3 leader_uuid: "0d00dab869714576ae199f6caf0c39d8" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8279b148b2454bfb90afa089240c8086" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 36693 } } peers { permanent_uuid: "0d00dab869714576ae199f6caf0c39d8" member_type: VOTER last_known_addr { host: "127.23.115.61" port: 45651 } attrs { promote: false } } peers { permanent_uuid: "294609dc6da34eefbb57726f025c5619" member_type: VOTER last_known_addr { host: "127.23.115.60" port: 33051 } attrs { promote: false } } }
I20260620 18:38:53.445952 31496 sys_catalog.cc:458] T 00000000000000000000000000000000 P 294609dc6da34eefbb57726f025c5619 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:38:53.447045 31489 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d00dab869714576ae199f6caf0c39d8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "0d00dab869714576ae199f6caf0c39d8" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8279b148b2454bfb90afa089240c8086" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 36693 } } peers { permanent_uuid: "0d00dab869714576ae199f6caf0c39d8" member_type: VOTER last_known_addr { host: "127.23.115.61" port: 45651 } attrs { promote: false } } peers { permanent_uuid: "294609dc6da34eefbb57726f025c5619" member_type: VOTER last_known_addr { host: "127.23.115.60" port: 33051 } attrs { promote: false } } }
I20260620 18:38:53.447374 31489 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d00dab869714576ae199f6caf0c39d8 [sys.catalog]: This master's current role is: LEADER
I20260620 18:38:53.447714 31496 sys_catalog.cc:455] T 00000000000000000000000000000000 P 294609dc6da34eefbb57726f025c5619 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "0d00dab869714576ae199f6caf0c39d8" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8279b148b2454bfb90afa089240c8086" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 36693 } } peers { permanent_uuid: "0d00dab869714576ae199f6caf0c39d8" member_type: VOTER last_known_addr { host: "127.23.115.61" port: 45651 } attrs { promote: false } } peers { permanent_uuid: "294609dc6da34eefbb57726f025c5619" member_type: VOTER last_known_addr { host: "127.23.115.60" port: 33051 } attrs { promote: false } } }
I20260620 18:38:53.447990 31496 sys_catalog.cc:458] T 00000000000000000000000000000000 P 294609dc6da34eefbb57726f025c5619 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:38:53.463575 31490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:53.455278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.3:44857 (local address 127.0.0.1:46082)
0620 18:38:53.455453 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:53.455472 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:53.455579 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:53.456349 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:53.456355 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:53.456385 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:38:53.456714 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:53.456726 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:53.458457 (+ 1731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:53.458464 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:53.460951 (+ 2487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:53.460967 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:53.462646 (+ 1679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:53.462654 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:53.462880 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:53.462924 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 18:38:53.463043 (+ 119us) client_negotiation.cc:240] Negotiation successful
0620 18:38:53.463203 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260620 18:38:53.465270 31501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:53.455415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46082 (local address 127.23.115.3:44857)
0620 18:38:53.455942 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:53.455947 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:53.455965 (+ 18us) server_negotiation.cc:407] Connection header received
0620 18:38:53.456051 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:53.456056 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:53.456177 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:53.456871 (+ 694us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:38:53.457055 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:53.458199 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:53.461169 (+ 2970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:53.462422 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:53.464543 (+ 2121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:53.464672 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:53.464729 (+ 57us) server_negotiation.cc:299] Negotiation successful
0620 18:38:53.464884 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":283,"threads_started":1}
I20260620 18:38:53.466209 31340 tablet_service.cc:1467] Processing CreateTablet for tablet 435f397ecc554ed7b117af4c106af960 (DEFAULT_TABLE table=test-table [id=c8de0586e1a84df090eeaf8f3149fb88]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:38:53.466699 31340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 435f397ecc554ed7b117af4c106af960. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:53.477012 31503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:53.455971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38120 (local address 127.23.115.2:33167)
0620 18:38:53.469896 (+ 13925us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:53.469910 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:53.469949 (+ 39us) server_negotiation.cc:407] Connection header received
0620 18:38:53.470156 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:53.470163 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:53.470239 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:53.470379 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:38:53.471065 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:53.472132 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:53.474694 (+ 2562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:53.475957 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:53.476076 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:53.476173 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:53.476565 (+ 392us) server_negotiation.cc:299] Negotiation successful
0620 18:38:53.476693 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13702,"thread_start_us":13558,"threads_started":1}
I20260620 18:38:53.477082 31502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:53.455529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.2:33167 (local address 127.0.0.1:38120)
0620 18:38:53.456083 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:53.456100 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:53.456274 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:53.470438 (+ 14164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:53.470444 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:53.470473 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:38:53.470867 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:53.470880 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:53.472286 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:53.472292 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:53.474491 (+ 2199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:53.474506 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:53.476290 (+ 1784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:53.476294 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:53.476411 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:53.476435 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 18:38:53.476515 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 18:38:53.476649 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"mutex_wait_us":1,"thread_start_us":879,"threads_started":1}
I20260620 18:38:53.478363 31199 tablet_service.cc:1467] Processing CreateTablet for tablet 435f397ecc554ed7b117af4c106af960 (DEFAULT_TABLE table=test-table [id=c8de0586e1a84df090eeaf8f3149fb88]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:38:53.478792 31199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 435f397ecc554ed7b117af4c106af960. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:53.478860 31490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:53.456709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.1:39357 (local address 127.0.0.1:43640)
0620 18:38:53.463840 (+ 7131us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:53.463861 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:53.463973 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:53.471504 (+ 7531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:53.471509 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:53.471536 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:38:53.471683 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:53.471692 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:53.473273 (+ 1581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:53.473294 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:53.475313 (+ 2019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:53.475328 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:53.478197 (+ 2869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:53.478203 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:53.478364 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:53.478397 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 18:38:53.478479 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 18:38:53.478603 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7022,"thread_start_us":12318,"threads_started":1}
I20260620 18:38:53.479097 31505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:53.456863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43640 (local address 127.23.115.1:39357)
0620 18:38:53.470070 (+ 13207us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:53.470076 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:53.470096 (+ 20us) server_negotiation.cc:407] Connection header received
0620 18:38:53.470167 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:53.470172 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:53.470623 (+ 451us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:53.471848 (+ 1225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:38:53.472041 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:53.473088 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:53.475528 (+ 2440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:53.478042 (+ 2514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:53.478184 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:53.478287 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:53.478536 (+ 249us) server_negotiation.cc:299] Negotiation successful
0620 18:38:53.478705 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13090,"thread_start_us":12884,"threads_started":1}
I20260620 18:38:53.479983 31051 tablet_service.cc:1467] Processing CreateTablet for tablet 435f397ecc554ed7b117af4c106af960 (DEFAULT_TABLE table=test-table [id=c8de0586e1a84df090eeaf8f3149fb88]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:38:53.480477 31051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 435f397ecc554ed7b117af4c106af960. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:53.481367 31506 tablet_bootstrap.cc:492] T 435f397ecc554ed7b117af4c106af960 P b8959fed0d23477fab84fd0a790447ce: Bootstrap starting.
I20260620 18:38:53.483104 31506 tablet_bootstrap.cc:654] T 435f397ecc554ed7b117af4c106af960 P b8959fed0d23477fab84fd0a790447ce: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:53.483429 31507 tablet_bootstrap.cc:492] T 435f397ecc554ed7b117af4c106af960 P 7b659e72cc154daf8c7b6de5cc3d0c29: Bootstrap starting.
I20260620 18:38:53.485237 31507 tablet_bootstrap.cc:654] T 435f397ecc554ed7b117af4c106af960 P 7b659e72cc154daf8c7b6de5cc3d0c29: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:53.488215 31507 tablet_bootstrap.cc:492] T 435f397ecc554ed7b117af4c106af960 P 7b659e72cc154daf8c7b6de5cc3d0c29: No bootstrap required, opened a new log
I20260620 18:38:53.488251 31506 tablet_bootstrap.cc:492] T 435f397ecc554ed7b117af4c106af960 P b8959fed0d23477fab84fd0a790447ce: No bootstrap required, opened a new log
I20260620 18:38:53.488402 31506 ts_tablet_manager.cc:1397] T 435f397ecc554ed7b117af4c106af960 P b8959fed0d23477fab84fd0a790447ce: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.007s
I20260620 18:38:53.488406 31507 ts_tablet_manager.cc:1397] T 435f397ecc554ed7b117af4c106af960 P 7b659e72cc154daf8c7b6de5cc3d0c29: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260620 18:38:53.488416 31510 tablet_bootstrap.cc:492] T 435f397ecc554ed7b117af4c106af960 P 74598bd1acf5422181769e91580ab45a: Bootstrap starting.
I20260620 18:38:53.489159 31506 raft_consensus.cc:348] T 435f397ecc554ed7b117af4c106af960 P b8959fed0d23477fab84fd0a790447ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8959fed0d23477fab84fd0a790447ce" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 44857 } } peers { permanent_uuid: "7b659e72cc154daf8c7b6de5cc3d0c29" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 33167 } } peers { permanent_uuid: "74598bd1acf5422181769e91580ab45a" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 39357 } }
I20260620 18:38:53.489159 31507 raft_consensus.cc:348] T 435f397ecc554ed7b117af4c106af960 P 7b659e72cc154daf8c7b6de5cc3d0c29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8959fed0d23477fab84fd0a790447ce" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 44857 } } peers { permanent_uuid: "7b659e72cc154daf8c7b6de5cc3d0c29" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 33167 } } peers { permanent_uuid: "74598bd1acf5422181769e91580ab45a" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 39357 } }
I20260620 18:38:53.489362 31506 raft_consensus.cc:374] T 435f397ecc554ed7b117af4c106af960 P b8959fed0d23477fab84fd0a790447ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:53.489387 31507 raft_consensus.cc:374] T 435f397ecc554ed7b117af4c106af960 P 7b659e72cc154daf8c7b6de5cc3d0c29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:53.489446 31506 raft_consensus.cc:729] T 435f397ecc554ed7b117af4c106af960 P b8959fed0d23477fab84fd0a790447ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8959fed0d23477fab84fd0a790447ce, State: Initialized, Role: FOLLOWER
I20260620 18:38:53.489462 31507 raft_consensus.cc:729] T 435f397ecc554ed7b117af4c106af960 P 7b659e72cc154daf8c7b6de5cc3d0c29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b659e72cc154daf8c7b6de5cc3d0c29, State: Initialized, Role: FOLLOWER
I20260620 18:38:53.489674 31507 consensus_queue.cc:260] T 435f397ecc554ed7b117af4c106af960 P 7b659e72cc154daf8c7b6de5cc3d0c29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8959fed0d23477fab84fd0a790447ce" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 44857 } } peers { permanent_uuid: "7b659e72cc154daf8c7b6de5cc3d0c29" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 33167 } } peers { permanent_uuid: "74598bd1acf5422181769e91580ab45a" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 39357 } }
I20260620 18:38:53.489683 31506 consensus_queue.cc:260] T 435f397ecc554ed7b117af4c106af960 P b8959fed0d23477fab84fd0a790447ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8959fed0d23477fab84fd0a790447ce" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 44857 } } peers { permanent_uuid: "7b659e72cc154daf8c7b6de5cc3d0c29" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 33167 } } peers { permanent_uuid: "74598bd1acf5422181769e91580ab45a" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 39357 } }
I20260620 18:38:53.489964 31510 tablet_bootstrap.cc:654] T 435f397ecc554ed7b117af4c106af960 P 74598bd1acf5422181769e91580ab45a: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:53.490487 31507 ts_tablet_manager.cc:1428] T 435f397ecc554ed7b117af4c106af960 P 7b659e72cc154daf8c7b6de5cc3d0c29: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 18:38:53.492112 31514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:53.491298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:36693 (local address 127.23.115.2:37067)
0620 18:38:53.491761 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:53.491916 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:36693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"threads_started":1}
I20260620 18:38:53.495666 31266 heartbeater.cc:503] Master 127.23.115.61:45651 was elected leader, sending a full tablet report...
I20260620 18:38:53.497056 31506 ts_tablet_manager.cc:1428] T 435f397ecc554ed7b117af4c106af960 P b8959fed0d23477fab84fd0a790447ce: Time spent starting tablet: real 0.009s user 0.003s sys 0.003s
I20260620 18:38:53.497947 31515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:53.496601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:36693 (local address 127.23.115.3:60303)
0620 18:38:53.497642 (+ 1041us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:53.497745 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:36693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":928,"thread_start_us":813,"threads_started":1}
I20260620 18:38:53.498445 31510 tablet_bootstrap.cc:492] T 435f397ecc554ed7b117af4c106af960 P 74598bd1acf5422181769e91580ab45a: No bootstrap required, opened a new log
I20260620 18:38:53.498646 31510 ts_tablet_manager.cc:1397] T 435f397ecc554ed7b117af4c106af960 P 74598bd1acf5422181769e91580ab45a: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.008s
I20260620 18:38:53.499490 31515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:53.498905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:36693 (local address 127.23.115.3:56239)
0620 18:38:53.499128 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:53.499207 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:36693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
I20260620 18:38:53.499521 31407 heartbeater.cc:503] Master 127.23.115.61:45651 was elected leader, sending a full tablet report...
I20260620 18:38:53.499498 31510 raft_consensus.cc:348] T 435f397ecc554ed7b117af4c106af960 P 74598bd1acf5422181769e91580ab45a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8959fed0d23477fab84fd0a790447ce" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 44857 } } peers { permanent_uuid: "7b659e72cc154daf8c7b6de5cc3d0c29" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 33167 } } peers { permanent_uuid: "74598bd1acf5422181769e91580ab45a" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 39357 } }
I20260620 18:38:53.499711 31510 raft_consensus.cc:374] T 435f397ecc554ed7b117af4c106af960 P 74598bd1acf5422181769e91580ab45a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:53.499807 31510 raft_consensus.cc:729] T 435f397ecc554ed7b117af4c106af960 P 74598bd1acf5422181769e91580ab45a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74598bd1acf5422181769e91580ab45a, State: Initialized, Role: FOLLOWER
I20260620 18:38:53.500056 31510 consensus_queue.cc:260] T 435f397ecc554ed7b117af4c106af960 P 74598bd1acf5422181769e91580ab45a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8959fed0d23477fab84fd0a790447ce" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 44857 } } peers { permanent_uuid: "7b659e72cc154daf8c7b6de5cc3d0c29" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 33167 } } peers { permanent_uuid: "74598bd1acf5422181769e91580ab45a" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 39357 } }
I20260620 18:38:53.500756 31510 ts_tablet_manager.cc:1428] T 435f397ecc554ed7b117af4c106af960 P 74598bd1acf5422181769e91580ab45a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 18:38:53.501937 31517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:53.501050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:36693 (local address 127.23.115.1:34845)
0620 18:38:53.501618 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:53.501746 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:36693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":377,"threads_started":1}
I20260620 18:38:53.503010 31118 heartbeater.cc:503] Master 127.23.115.61:45651 was elected leader, sending a full tablet report...
I20260620 18:38:53.503352 31517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:53.502994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:36693 (local address 127.23.115.1:43463)
0620 18:38:53.503159 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:53.503240 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:36693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 18:38:53.516419 31119 tablet.cc:2369] T 435f397ecc554ed7b117af4c106af960 P 74598bd1acf5422181769e91580ab45a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:38:53.631410 31512 raft_consensus.cc:482] T 435f397ecc554ed7b117af4c106af960 P b8959fed0d23477fab84fd0a790447ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:38:53.631649 31512 raft_consensus.cc:504] T 435f397ecc554ed7b117af4c106af960 P b8959fed0d23477fab84fd0a790447ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8959fed0d23477fab84fd0a790447ce" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 44857 } } peers { permanent_uuid: "7b659e72cc154daf8c7b6de5cc3d0c29" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 33167 } } peers { permanent_uuid: "74598bd1acf5422181769e91580ab45a" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 39357 } }
I20260620 18:38:53.633258 31512 leader_election.cc:288] T 435f397ecc554ed7b117af4c106af960 P b8959fed0d23477fab84fd0a790447ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b659e72cc154daf8c7b6de5cc3d0c29 (127.23.115.2:33167), 74598bd1acf5422181769e91580ab45a (127.23.115.1:39357)
I20260620 18:38:53.639616 31503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:53.633623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.3:41395 (local address 127.23.115.2:33167)
0620 18:38:53.633782 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:53.633787 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:53.633865 (+ 78us) server_negotiation.cc:407] Connection header received
0620 18:38:53.634092 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:53.634098 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:53.634175 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:53.634282 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:38:53.634965 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:53.635878 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:53.637797 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:53.638750 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:53.638918 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:53.639019 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:53.639315 (+ 296us) server_negotiation.cc:299] Negotiation successful
0620 18:38:53.639413 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 18:38:53.639689 31515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:53.633552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.2:33167 (local address 127.23.115.3:41395)
0620 18:38:53.633743 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:53.633777 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:53.633942 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:53.634433 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:53.634438 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:53.634473 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:38:53.634766 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:53.634780 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:53.636011 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:53.636017 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:53.637635 (+ 1618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:53.637648 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:53.638965 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:53.638972 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:53.639124 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:53.639156 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 18:38:53.639254 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 18:38:53.639400 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260620 18:38:53.639954 31505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:53.634065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.3:35421 (local address 127.23.115.1:39357)
0620 18:38:53.634329 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:53.634336 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:53.634711 (+ 375us) server_negotiation.cc:407] Connection header received
0620 18:38:53.634851 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:53.634858 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:53.634966 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:53.635050 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:38:53.635580 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:53.636471 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:53.638225 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:53.639075 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:53.639210 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:53.639277 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:53.639598 (+ 321us) server_negotiation.cc:299] Negotiation successful
0620 18:38:53.639725 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 18:38:53.640214 31518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:53.633967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.1:39357 (local address 127.23.115.3:35421)
0620 18:38:53.634613 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:53.634626 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:53.634732 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:53.635097 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:53.635102 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:53.635127 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:38:53.635430 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:53.635441 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:53.636602 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:53.636605 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:53.638030 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:53.638044 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:53.639247 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:53.639253 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:53.639452 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:53.639482 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 18:38:53.639564 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 18:38:53.639680 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"mutex_wait_us":1,"thread_start_us":345,"threads_started":1}
I20260620 18:38:53.640522 31219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "435f397ecc554ed7b117af4c106af960" candidate_uuid: "b8959fed0d23477fab84fd0a790447ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b659e72cc154daf8c7b6de5cc3d0c29" is_pre_election: true
I20260620 18:38:53.640801 31219 raft_consensus.cc:2454] T 435f397ecc554ed7b117af4c106af960 P 7b659e72cc154daf8c7b6de5cc3d0c29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8959fed0d23477fab84fd0a790447ce in term 0.
I20260620 18:38:53.641000 31071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "435f397ecc554ed7b117af4c106af960" candidate_uuid: "b8959fed0d23477fab84fd0a790447ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74598bd1acf5422181769e91580ab45a" is_pre_election: true
I20260620 18:38:53.641296 31294 leader_election.cc:302] T 435f397ecc554ed7b117af4c106af960 P b8959fed0d23477fab84fd0a790447ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b659e72cc154daf8c7b6de5cc3d0c29, b8959fed0d23477fab84fd0a790447ce; no voters:
I20260620 18:38:53.641593 31071 raft_consensus.cc:2454] T 435f397ecc554ed7b117af4c106af960 P 74598bd1acf5422181769e91580ab45a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8959fed0d23477fab84fd0a790447ce in term 0.
I20260620 18:38:53.641770 31512 raft_consensus.cc:2781] T 435f397ecc554ed7b117af4c106af960 P b8959fed0d23477fab84fd0a790447ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:38:53.641908 31512 raft_consensus.cc:482] T 435f397ecc554ed7b117af4c106af960 P b8959fed0d23477fab84fd0a790447ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:38:53.642063 31512 raft_consensus.cc:3037] T 435f397ecc554ed7b117af4c106af960 P b8959fed0d23477fab84fd0a790447ce [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:53.644428 31512 raft_consensus.cc:504] T 435f397ecc554ed7b117af4c106af960 P b8959fed0d23477fab84fd0a790447ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8959fed0d23477fab84fd0a790447ce" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 44857 } } peers { permanent_uuid: "7b659e72cc154daf8c7b6de5cc3d0c29" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 33167 } } peers { permanent_uuid: "74598bd1acf5422181769e91580ab45a" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 39357 } }
I20260620 18:38:53.645190 31512 leader_election.cc:288] T 435f397ecc554ed7b117af4c106af960 P b8959fed0d23477fab84fd0a790447ce [CANDIDATE]: Term 1 election: Requested vote from peers 7b659e72cc154daf8c7b6de5cc3d0c29 (127.23.115.2:33167), 74598bd1acf5422181769e91580ab45a (127.23.115.1:39357)
I20260620 18:38:53.645531 31219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "435f397ecc554ed7b117af4c106af960" candidate_uuid: "b8959fed0d23477fab84fd0a790447ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b659e72cc154daf8c7b6de5cc3d0c29"
I20260620 18:38:53.645601 31071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "435f397ecc554ed7b117af4c106af960" candidate_uuid: "b8959fed0d23477fab84fd0a790447ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74598bd1acf5422181769e91580ab45a"
I20260620 18:38:53.645692 31219 raft_consensus.cc:3037] T 435f397ecc554ed7b117af4c106af960 P 7b659e72cc154daf8c7b6de5cc3d0c29 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:53.645800 31071 raft_consensus.cc:3037] T 435f397ecc554ed7b117af4c106af960 P 74598bd1acf5422181769e91580ab45a [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:53.647763 31219 raft_consensus.cc:2454] T 435f397ecc554ed7b117af4c106af960 P 7b659e72cc154daf8c7b6de5cc3d0c29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8959fed0d23477fab84fd0a790447ce in term 1.
I20260620 18:38:53.647842 31071 raft_consensus.cc:2454] T 435f397ecc554ed7b117af4c106af960 P 74598bd1acf5422181769e91580ab45a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8959fed0d23477fab84fd0a790447ce in term 1.
I20260620 18:38:53.648175 31294 leader_election.cc:302] T 435f397ecc554ed7b117af4c106af960 P b8959fed0d23477fab84fd0a790447ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b659e72cc154daf8c7b6de5cc3d0c29, b8959fed0d23477fab84fd0a790447ce; no voters:
I20260620 18:38:53.648468 31512 raft_consensus.cc:2781] T 435f397ecc554ed7b117af4c106af960 P b8959fed0d23477fab84fd0a790447ce [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:38:53.648943 31512 raft_consensus.cc:686] T 435f397ecc554ed7b117af4c106af960 P b8959fed0d23477fab84fd0a790447ce [term 1 LEADER]: Becoming Leader. State: Replica: b8959fed0d23477fab84fd0a790447ce, State: Running, Role: LEADER
I20260620 18:38:53.649560 31512 consensus_queue.cc:237] T 435f397ecc554ed7b117af4c106af960 P b8959fed0d23477fab84fd0a790447ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8959fed0d23477fab84fd0a790447ce" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 44857 } } peers { permanent_uuid: "7b659e72cc154daf8c7b6de5cc3d0c29" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 33167 } } peers { permanent_uuid: "74598bd1acf5422181769e91580ab45a" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 39357 } }
I20260620 18:38:53.649931 31518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:53.649517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:36693 (local address 127.23.115.3:60913)
0620 18:38:53.649664 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:53.649779 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:36693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260620 18:38:53.652994 30746 catalog_manager.cc:5495] T 435f397ecc554ed7b117af4c106af960 P b8959fed0d23477fab84fd0a790447ce reported cstate change: term changed from 0 to 1, leader changed from <none> to b8959fed0d23477fab84fd0a790447ce (127.23.115.3). New cstate: current_term: 1 leader_uuid: "b8959fed0d23477fab84fd0a790447ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8959fed0d23477fab84fd0a790447ce" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 44857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7b659e72cc154daf8c7b6de5cc3d0c29" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 33167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74598bd1acf5422181769e91580ab45a" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 39357 } health_report { overall_health: UNKNOWN } } }
I20260620 18:38:53.706956 31501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:53.700856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46098 (local address 127.23.115.3:44857)
0620 18:38:53.701089 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:53.701094 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:53.701113 (+ 19us) server_negotiation.cc:407] Connection header received
0620 18:38:53.701206 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:53.701212 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:53.701314 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:53.701473 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:38:53.701821 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:53.702980 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:53.704232 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:53.704904 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:53.705012 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:53.705557 (+ 545us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:38:53.706235 (+ 678us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:38:53.706352 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:53.706570 (+ 218us) server_negotiation.cc:299] Negotiation successful
0620 18:38:53.706656 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260620 18:38:53.713491 31219 raft_consensus.cc:1261] T 435f397ecc554ed7b117af4c106af960 P 7b659e72cc154daf8c7b6de5cc3d0c29 [term 1 FOLLOWER]: Refusing update from remote peer b8959fed0d23477fab84fd0a790447ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:38:53.713785 31071 raft_consensus.cc:1261] T 435f397ecc554ed7b117af4c106af960 P 74598bd1acf5422181769e91580ab45a [term 1 FOLLOWER]: Refusing update from remote peer b8959fed0d23477fab84fd0a790447ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:38:53.714358 31512 consensus_queue.cc:1035] T 435f397ecc554ed7b117af4c106af960 P b8959fed0d23477fab84fd0a790447ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b659e72cc154daf8c7b6de5cc3d0c29" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 33167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:38:53.714737 31519 consensus_queue.cc:1035] T 435f397ecc554ed7b117af4c106af960 P b8959fed0d23477fab84fd0a790447ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "74598bd1acf5422181769e91580ab45a" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 39357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:38:53.717061 31514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:53.716556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:36693 (local address 127.23.115.2:60069)
0620 18:38:53.716772 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:53.716881 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:36693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260620 18:38:53.720587 31517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:53.719974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:36693 (local address 127.23.115.1:38611)
0620 18:38:53.720277 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:53.720381 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:36693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192}
I20260620 18:38:53.728125 31517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:53.727550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:36693 (local address 127.23.115.1:41393)
0620 18:38:53.727848 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:53.727946 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:36693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142}
I20260620 18:38:53.736616 31518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:53.735821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:36693 (local address 127.23.115.3:60311)
0620 18:38:53.736296 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:53.736391 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:36693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353}
I20260620 18:38:53.751592 30746 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34588:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:38:53.751834 30746 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34588:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:38:53.754789 31518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:53.753584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:36693 (local address 127.23.115.3:37327)
0620 18:38:53.754497 (+ 913us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:53.754595 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:36693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":785}
I20260620 18:38:53.755556 31514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:53.755140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:36693 (local address 127.23.115.2:39047)
0620 18:38:53.755317 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:53.755402 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:36693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260620 18:38:53.760699 30746 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0d00dab869714576ae199f6caf0c39d8: Sending DeleteTablet for 3 replicas of tablet 435f397ecc554ed7b117af4c106af960
I20260620 18:38:53.761598 31340 tablet_service.cc:1514] Processing DeleteTablet for tablet 435f397ecc554ed7b117af4c106af960 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:38:53 UTC) from {username='oryx', principal='oryx/127.23.115.61@KRBTEST.COM'} at 127.0.0.1:46082
I20260620 18:38:53.762022 31199 tablet_service.cc:1514] Processing DeleteTablet for tablet 435f397ecc554ed7b117af4c106af960 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:38:53 UTC) from {username='oryx', principal='oryx/127.23.115.61@KRBTEST.COM'} at 127.0.0.1:38120
I20260620 18:38:53.763113 31051 tablet_service.cc:1514] Processing DeleteTablet for tablet 435f397ecc554ed7b117af4c106af960 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:38:53 UTC) from {username='oryx', principal='oryx/127.23.115.61@KRBTEST.COM'} at 127.0.0.1:43640
I20260620 18:38:53.763965 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 30924
I20260620 18:38:53.766777 31534 tablet_replica.cc:330] T 435f397ecc554ed7b117af4c106af960 P 7b659e72cc154daf8c7b6de5cc3d0c29: stopping tablet replica
I20260620 18:38:53.767051 31534 raft_consensus.cc:2229] T 435f397ecc554ed7b117af4c106af960 P 7b659e72cc154daf8c7b6de5cc3d0c29 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:38:53.767426 31534 raft_consensus.cc:2258] T 435f397ecc554ed7b117af4c106af960 P 7b659e72cc154daf8c7b6de5cc3d0c29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:38:53.768455 31535 tablet_replica.cc:330] T 435f397ecc554ed7b117af4c106af960 P 74598bd1acf5422181769e91580ab45a: stopping tablet replica
I20260620 18:38:53.768639 31535 raft_consensus.cc:2229] T 435f397ecc554ed7b117af4c106af960 P 74598bd1acf5422181769e91580ab45a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:38:53.768929 31535 raft_consensus.cc:2258] T 435f397ecc554ed7b117af4c106af960 P 74598bd1acf5422181769e91580ab45a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:38:53.770345 31534 ts_tablet_manager.cc:1905] T 435f397ecc554ed7b117af4c106af960 P 7b659e72cc154daf8c7b6de5cc3d0c29: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:38:53.770843 31535 ts_tablet_manager.cc:1905] T 435f397ecc554ed7b117af4c106af960 P 74598bd1acf5422181769e91580ab45a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:38:53.776677 31535 ts_tablet_manager.cc:1918] T 435f397ecc554ed7b117af4c106af960 P 74598bd1acf5422181769e91580ab45a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:38:53.776820 31535 log.cc:1196] T 435f397ecc554ed7b117af4c106af960 P 74598bd1acf5422181769e91580ab45a: Deleting WAL directory at /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/ts-0/wal/wals/435f397ecc554ed7b117af4c106af960
I20260620 18:38:53.777585 31533 tablet_replica.cc:330] T 435f397ecc554ed7b117af4c106af960 P b8959fed0d23477fab84fd0a790447ce: stopping tablet replica
I20260620 18:38:53.777822 31533 raft_consensus.cc:2229] T 435f397ecc554ed7b117af4c106af960 P b8959fed0d23477fab84fd0a790447ce [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:38:53.778560 31535 ts_tablet_manager.cc:1939] T 435f397ecc554ed7b117af4c106af960 P 74598bd1acf5422181769e91580ab45a: Deleting consensus metadata
I20260620 18:38:53.778540 31533 raft_consensus.cc:2258] T 435f397ecc554ed7b117af4c106af960 P b8959fed0d23477fab84fd0a790447ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:38:53.779898 31533 ts_tablet_manager.cc:1905] T 435f397ecc554ed7b117af4c106af960 P b8959fed0d23477fab84fd0a790447ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:38:53.779881 30733 catalog_manager.cc:4841] TS 74598bd1acf5422181769e91580ab45a (127.23.115.1:39357): tablet 435f397ecc554ed7b117af4c106af960 (table test-table [id=c8de0586e1a84df090eeaf8f3149fb88]) successfully deleted
I20260620 18:38:53.785458 31534 ts_tablet_manager.cc:1918] T 435f397ecc554ed7b117af4c106af960 P 7b659e72cc154daf8c7b6de5cc3d0c29: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:38:53.785635 31534 log.cc:1196] T 435f397ecc554ed7b117af4c106af960 P 7b659e72cc154daf8c7b6de5cc3d0c29: Deleting WAL directory at /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/ts-1/wal/wals/435f397ecc554ed7b117af4c106af960
I20260620 18:38:53.786417 31534 ts_tablet_manager.cc:1939] T 435f397ecc554ed7b117af4c106af960 P 7b659e72cc154daf8c7b6de5cc3d0c29: Deleting consensus metadata
I20260620 18:38:53.786486 31533 ts_tablet_manager.cc:1918] T 435f397ecc554ed7b117af4c106af960 P b8959fed0d23477fab84fd0a790447ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:38:53.786629 31533 log.cc:1196] T 435f397ecc554ed7b117af4c106af960 P b8959fed0d23477fab84fd0a790447ce: Deleting WAL directory at /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980676253068-24012-0/minicluster-data/ts-2/wal/wals/435f397ecc554ed7b117af4c106af960
I20260620 18:38:53.787181 31533 ts_tablet_manager.cc:1939] T 435f397ecc554ed7b117af4c106af960 P b8959fed0d23477fab84fd0a790447ce: Deleting consensus metadata
I20260620 18:38:53.788028 30733 catalog_manager.cc:4841] TS 7b659e72cc154daf8c7b6de5cc3d0c29 (127.23.115.2:33167): tablet 435f397ecc554ed7b117af4c106af960 (table test-table [id=c8de0586e1a84df090eeaf8f3149fb88]) successfully deleted
I20260620 18:38:53.788405 30735 catalog_manager.cc:4841] TS b8959fed0d23477fab84fd0a790447ce (127.23.115.3:44857): tablet 435f397ecc554ed7b117af4c106af960 (table test-table [id=c8de0586e1a84df090eeaf8f3149fb88]) successfully deleted
I20260620 18:38:53.792518 31505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:53.764946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43648 (local address 127.23.115.1:39357)
0620 18:38:53.765112 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:53.765118 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:53.769725 (+ 4607us) server_negotiation.cc:407] Connection header received
0620 18:38:53.769803 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:53.769809 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:53.769895 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:53.770010 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:53.777035 (+ 7025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:53.778044 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:53.780197 (+ 2153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:53.780824 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:53.780969 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:53.785063 (+ 4094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:53.785083 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:53.785104 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:53.785153 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:53.789481 (+ 4328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:53.790203 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:53.790212 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:53.790216 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:53.790393 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:53.790824 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:53.790829 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:53.790833 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:53.791344 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:53.791636 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:53.791895 (+ 259us) server_negotiation.cc:299] Negotiation successful
0620 18:38:53.792141 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 18:38:53.793200 31111 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:38:53.852626 31490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:53.852096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:36693 (local address 127.0.0.1:47762)
0620 18:38:53.852288 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:53.852414 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:36693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260620 18:38:53.853915 31490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:53.853424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:36693 (local address 127.0.0.1:47770)
0620 18:38:53.853698 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:53.853780 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:36693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169}
I20260620 18:38:53.917445 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 30924
I20260620 18:38:53.945156 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 31128
I20260620 18:38:53.958673 31503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:53.946059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38134 (local address 127.23.115.2:33167)
0620 18:38:53.946275 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:53.946281 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:53.946305 (+ 24us) server_negotiation.cc:407] Connection header received
0620 18:38:53.946377 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:53.946383 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:53.946468 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:53.946594 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:53.947869 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:53.948809 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:53.950055 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:53.950457 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:53.950562 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:53.953185 (+ 2623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:53.953219 (+ 34us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:53.953238 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:53.953278 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:53.956119 (+ 2841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:53.956747 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:53.956754 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:53.956756 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:53.956880 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:53.957221 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:53.957225 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:53.957227 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:53.957564 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:53.957860 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:53.958132 (+ 272us) server_negotiation.cc:299] Negotiation successful
0620 18:38:53.958347 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260620 18:38:53.959565 31259 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:38:54.069456 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 31128
I20260620 18:38:54.091095 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 31277
I20260620 18:38:54.105456 31501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:54.092143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46110 (local address 127.23.115.3:44857)
0620 18:38:54.092347 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:54.092353 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:54.092375 (+ 22us) server_negotiation.cc:407] Connection header received
0620 18:38:54.092446 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:54.092452 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:54.092554 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:54.092684 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:54.093973 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:54.095279 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:54.096517 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:54.096957 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:54.097072 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:54.099580 (+ 2508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:54.099594 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:54.099609 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:54.099650 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:54.102467 (+ 2817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:54.103206 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:54.103219 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:54.103222 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:54.103396 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:54.103752 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:54.103757 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:54.103761 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:54.104226 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:54.104559 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:54.104848 (+ 289us) server_negotiation.cc:299] Negotiation successful
0620 18:38:54.105081 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 18:38:54.106397 31400 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:38:54.233873 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 31277
I20260620 18:38:54.250440 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 30610
I20260620 18:38:54.251538 30778 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:38:54.372210 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 30610
W20260620 18:38:54.389986 30922 connection.cc:535] server connection from 127.0.0.1:38390 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 18:38:54.390241 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 30711
I20260620 18:38:54.391269 30967 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:38:54.516698 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 30711
2026-06-20T18:38:54Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (11423 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980676253068-24012-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:38:54 dist-test-slave-d645 krb5kdc[31545](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:38:54 dist-test-slave-d645 krb5kdc[31545](info): set up 2 sockets
Jun 20 18:38:54 dist-test-slave-d645 krb5kdc[31545](info): commencing operation
krb5kdc: starting...
W20260620 18:38:55.543792 24012 mini_kdc.cc:121] Time spent starting KDC: real 0.986s user 0.004s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980676253068-24012-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980676253068-24012-0/krb5kdc/test-user.keytab.
Jun 20 18:38:55 dist-test-slave-d645 krb5kdc[31545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980735, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:38:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:38:55Z Disabled control of system clock
WARNING: no policy specified for oryx/127.23.115.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.115.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.115.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980676253068-24012-0/krb5kdc/oryx_127.23.115.62.keytab.
Entry for principal oryx/127.23.115.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980676253068-24012-0/krb5kdc/oryx_127.23.115.62.keytab.
I20260620 18:38:55.747850 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980676253068-24012-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980676253068-24012-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980676253068-24012-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.115.62:38311
--webserver_interface=127.23.115.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980676253068-24012-0/krb5kdc/oryx_127.23.115.62.keytab
--principal=oryx/127.23.115.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:45287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.115.62:38311
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:38:56.061491 31558 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:56.061801 31558 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:56.061945 31558 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:56.069301 31558 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:38:56.069414 31558 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 18:38:56.069482 31558 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:56.069568 31558 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:56.069641 31558 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:38:56.069722 31558 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:38:56.079630 31558 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:45287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980676253068-24012-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980676253068-24012-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.115.62:38311
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980676253068-24012-0/krb5kdc/oryx_127.23.115.62.keytab
--principal=oryx/127.23.115.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.115.62:38311
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.115.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980676253068-24012-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:38:56.080927 31558 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:56.082190 31558 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:56.094633 31566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:56.094673 31564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:56.094547 31563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:56.100495 31558 server_base.cc:1034] running on GCE node
I20260620 18:38:56.101300 31558 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:56.102751 31558 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:56.104013 31558 hybrid_clock.cc:648] HybridClock initialized: now 1781980736103938 us; error 54 us; skew 500 ppm
I20260620 18:38:56.104331 31558 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:38:56 dist-test-slave-d645 krb5kdc[31545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980736, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.115.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:56.107328 31558 init.cc:377] Logged in from keytab as oryx/127.23.115.62@KRBTEST.COM (short username oryx)
I20260620 18:38:56.108785 31558 webserver.cc:460] Webserver started at http://127.23.115.62:33169/ using document root <none> and password file <none>
I20260620 18:38:56.109441 31558 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:56.109524 31558 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:56.109795 31558 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:38:56.112620 31558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980676253068-24012-0/minicluster-data/master-0/data/instance:
uuid: "522d0826a3f64949a4b30e203c4b3a74"
format_stamp: "Formatted at 2026-06-20 18:38:56 on dist-test-slave-d645"
server_key: "a5bcc39b3ed45a96289ff1133f11dc56"
server_key_iv: "03688f5d1c3f9524890e0df348af7446"
server_key_version: "encryptionkey@0"
I20260620 18:38:56.113334 31558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980676253068-24012-0/minicluster-data/master-0/wal/instance:
uuid: "522d0826a3f64949a4b30e203c4b3a74"
format_stamp: "Formatted at 2026-06-20 18:38:56 on dist-test-slave-d645"
server_key: "a5bcc39b3ed45a96289ff1133f11dc56"
server_key_iv: "03688f5d1c3f9524890e0df348af7446"
server_key_version: "encryptionkey@0"
I20260620 18:38:56.118093 31558 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 18:38:56.121460 31573 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:56.122790 31558 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:38:56.122990 31558 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980676253068-24012-0/minicluster-data/master-0/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980676253068-24012-0/minicluster-data/master-0/wal
uuid: "522d0826a3f64949a4b30e203c4b3a74"
format_stamp: "Formatted at 2026-06-20 18:38:56 on dist-test-slave-d645"
server_key: "a5bcc39b3ed45a96289ff1133f11dc56"
server_key_iv: "03688f5d1c3f9524890e0df348af7446"
server_key_version: "encryptionkey@0"
I20260620 18:38:56.123178 31558 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980676253068-24012-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980676253068-24012-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980676253068-24012-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:56.138180 31558 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:56.142052 31558 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:56.142263 31558 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:56.162248 31558 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.62:38311
I20260620 18:38:56.162287 31634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.62:38311 every 8 connection(s)
I20260620 18:38:56.163597 31558 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
I20260620 18:38:56.167433 31635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:56.167654 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 31558
I20260620 18:38:56.167850 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980676253068-24012-0/minicluster-data/master-0/wal/instance
I20260620 18:38:56.168649 24012 external_mini_cluster.cc:1428] Setting key 8f96e9b114fe70bc02b5db39153bf67c
I20260620 18:38:56.181239 31635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 522d0826a3f64949a4b30e203c4b3a74: Bootstrap starting.
Jun 20 18:38:56 dist-test-slave-d645 krb5kdc[31545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980735, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM
I20260620 18:38:56.184978 31635 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 522d0826a3f64949a4b30e203c4b3a74: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:56.186005 31635 log.cc:824] T 00000000000000000000000000000000 P 522d0826a3f64949a4b30e203c4b3a74: Log is configured to *not* fsync() on all Append() calls
I20260620 18:38:56.189114 31635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 522d0826a3f64949a4b30e203c4b3a74: No bootstrap required, opened a new log
I20260620 18:38:56.192605 31638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:56.170590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38652 (local address 127.23.115.62:38311)
0620 18:38:56.171364 (+ 774us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:56.171374 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:56.171415 (+ 41us) server_negotiation.cc:407] Connection header received
0620 18:38:56.173507 (+ 2092us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:56.173534 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:56.173923 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:56.174244 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:56.175998 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:56.177270 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:56.179077 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:56.179781 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:56.179933 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:56.184938 (+ 5005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:56.184948 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:56.184969 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:56.185017 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:56.188379 (+ 3362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:56.189228 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:56.189237 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:56.189246 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:56.189434 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:56.189890 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:56.189894 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:56.189896 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:56.190449 (+ 553us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:56.190685 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:56.191163 (+ 478us) server_negotiation.cc:299] Negotiation successful
0620 18:38:56.191602 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":381,"threads_started":1}
I20260620 18:38:56.196892 31635 raft_consensus.cc:348] T 00000000000000000000000000000000 P 522d0826a3f64949a4b30e203c4b3a74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "522d0826a3f64949a4b30e203c4b3a74" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 38311 } }
I20260620 18:38:56.197214 31635 raft_consensus.cc:374] T 00000000000000000000000000000000 P 522d0826a3f64949a4b30e203c4b3a74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:56.197319 31635 raft_consensus.cc:729] T 00000000000000000000000000000000 P 522d0826a3f64949a4b30e203c4b3a74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 522d0826a3f64949a4b30e203c4b3a74, State: Initialized, Role: FOLLOWER
I20260620 18:38:56.197829 31635 consensus_queue.cc:260] T 00000000000000000000000000000000 P 522d0826a3f64949a4b30e203c4b3a74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "522d0826a3f64949a4b30e203c4b3a74" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 38311 } }
I20260620 18:38:56.198020 31635 raft_consensus.cc:388] T 00000000000000000000000000000000 P 522d0826a3f64949a4b30e203c4b3a74 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:38:56.198137 31635 raft_consensus.cc:482] T 00000000000000000000000000000000 P 522d0826a3f64949a4b30e203c4b3a74 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:38:56.198287 31635 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 522d0826a3f64949a4b30e203c4b3a74 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:56.201167 31635 raft_consensus.cc:504] T 00000000000000000000000000000000 P 522d0826a3f64949a4b30e203c4b3a74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "522d0826a3f64949a4b30e203c4b3a74" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 38311 } }
I20260620 18:38:56.201730 31635 leader_election.cc:302] T 00000000000000000000000000000000 P 522d0826a3f64949a4b30e203c4b3a74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 522d0826a3f64949a4b30e203c4b3a74; no voters:
I20260620 18:38:56.202328 31635 leader_election.cc:288] T 00000000000000000000000000000000 P 522d0826a3f64949a4b30e203c4b3a74 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:38:56.202461 31640 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 522d0826a3f64949a4b30e203c4b3a74 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:38:56.203179 31640 raft_consensus.cc:686] T 00000000000000000000000000000000 P 522d0826a3f64949a4b30e203c4b3a74 [term 1 LEADER]: Becoming Leader. State: Replica: 522d0826a3f64949a4b30e203c4b3a74, State: Running, Role: LEADER
I20260620 18:38:56.203712 31640 consensus_queue.cc:237] T 00000000000000000000000000000000 P 522d0826a3f64949a4b30e203c4b3a74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "522d0826a3f64949a4b30e203c4b3a74" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 38311 } }
I20260620 18:38:56.204363 31635 sys_catalog.cc:564] T 00000000000000000000000000000000 P 522d0826a3f64949a4b30e203c4b3a74 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:38:56.206254 31642 sys_catalog.cc:455] T 00000000000000000000000000000000 P 522d0826a3f64949a4b30e203c4b3a74 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 522d0826a3f64949a4b30e203c4b3a74. Latest consensus state: current_term: 1 leader_uuid: "522d0826a3f64949a4b30e203c4b3a74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "522d0826a3f64949a4b30e203c4b3a74" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 38311 } } }
I20260620 18:38:56.206673 31642 sys_catalog.cc:458] T 00000000000000000000000000000000 P 522d0826a3f64949a4b30e203c4b3a74 [sys.catalog]: This master's current role is: LEADER
I20260620 18:38:56.206283 31641 sys_catalog.cc:455] T 00000000000000000000000000000000 P 522d0826a3f64949a4b30e203c4b3a74 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "522d0826a3f64949a4b30e203c4b3a74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "522d0826a3f64949a4b30e203c4b3a74" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 38311 } } }
I20260620 18:38:56.210327 31641 sys_catalog.cc:458] T 00000000000000000000000000000000 P 522d0826a3f64949a4b30e203c4b3a74 [sys.catalog]: This master's current role is: LEADER
I20260620 18:38:56.211920 31646 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:38:56.217768 31646 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:38:56.226310 31646 catalog_manager.cc:1347] Generated new cluster ID: 093fdc0066714611a11a052e8b01aa4b
I20260620 18:38:56.226425 31646 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:38:56.236650 31646 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:38:56.238531 31646 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:38:56.245887 31646 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 522d0826a3f64949a4b30e203c4b3a74: Generated new TSK 0
I20260620 18:38:56.246814 31646 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.23.115.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.115.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.115.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980676253068-24012-0/krb5kdc/oryx_127.23.115.1.keytab.
Entry for principal oryx/127.23.115.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980676253068-24012-0/krb5kdc/oryx_127.23.115.1.keytab.
I20260620 18:38:56.300405 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980676253068-24012-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980676253068-24012-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980676253068-24012-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980676253068-24012-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.115.1:0
--local_ip_for_outbound_sockets=127.23.115.1
--webserver_interface=127.23.115.1
--webserver_port=0
--tserver_master_addrs=127.23.115.62:38311
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980676253068-24012-0/krb5kdc/oryx_127.23.115.1.keytab
--principal=oryx/127.23.115.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:45287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:38:56.615223 31661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:56.615506 31661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:56.615636 31661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:56.622891 31661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:56.623031 31661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:56.623163 31661 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 18:38:56.623270 31661 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.115.1
I20260620 18:38:56.634116 31661 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:45287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980676253068-24012-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980676253068-24012-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980676253068-24012-0/krb5kdc/oryx_127.23.115.1.keytab
--principal=oryx/127.23.115.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.115.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980676253068-24012-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.115.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.115.62:38311
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.115.1
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980676253068-24012-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:38:56.635452 31661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:56.636691 31661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:56.649076 31667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:56.649269 31669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:56.649528 31666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:56.649816 31661 server_base.cc:1034] running on GCE node
I20260620 18:38:56.650372 31661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:56.651270 31661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:56.652524 31661 hybrid_clock.cc:648] HybridClock initialized: now 1781980736652452 us; error 44 us; skew 500 ppm
I20260620 18:38:56.652805 31661 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:38:56 dist-test-slave-d645 krb5kdc[31545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980736, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.115.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:56.656004 31661 init.cc:377] Logged in from keytab as oryx/127.23.115.1@KRBTEST.COM (short username oryx)
I20260620 18:38:56.657471 31661 webserver.cc:460] Webserver started at http://127.23.115.1:41865/ using document root <none> and password file <none>
I20260620 18:38:56.658157 31661 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:56.658257 31661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:56.658556 31661 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:38:56.661367 31661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980676253068-24012-0/minicluster-data/ts-0/data/instance:
uuid: "2dac9332ea0345ceb33c40b1779316b2"
format_stamp: "Formatted at 2026-06-20 18:38:56 on dist-test-slave-d645"
server_key: "7c4e4c1f2ebedd624079b8bd3a479306"
server_key_iv: "63433983d216b076353f621309f689f1"
server_key_version: "encryptionkey@0"
I20260620 18:38:56.662125 31661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980676253068-24012-0/minicluster-data/ts-0/wal/instance:
uuid: "2dac9332ea0345ceb33c40b1779316b2"
format_stamp: "Formatted at 2026-06-20 18:38:56 on dist-test-slave-d645"
server_key: "7c4e4c1f2ebedd624079b8bd3a479306"
server_key_iv: "63433983d216b076353f621309f689f1"
server_key_version: "encryptionkey@0"
I20260620 18:38:56.667583 31661 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 18:38:56.671676 31676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:56.673019 31661 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:38:56.673178 31661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980676253068-24012-0/minicluster-data/ts-0/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980676253068-24012-0/minicluster-data/ts-0/wal
uuid: "2dac9332ea0345ceb33c40b1779316b2"
format_stamp: "Formatted at 2026-06-20 18:38:56 on dist-test-slave-d645"
server_key: "7c4e4c1f2ebedd624079b8bd3a479306"
server_key_iv: "63433983d216b076353f621309f689f1"
server_key_version: "encryptionkey@0"
I20260620 18:38:56.673405 31661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980676253068-24012-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980676253068-24012-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980676253068-24012-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:56.687913 31661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:56.692624 31661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:56.692883 31661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:56.696028 31661 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:38:56.696111 31661 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:56.696245 31661 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:38:56.696311 31661 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 18:38:56 dist-test-slave-d645 krb5kdc[31545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980736, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.115.1@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM
I20260620 18:38:56.721293 31685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:56.696807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.1:44467 (local address 127.23.115.62:38311)
0620 18:38:56.697275 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:56.697282 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:56.698468 (+ 1186us) server_negotiation.cc:407] Connection header received
0620 18:38:56.700761 (+ 2293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:56.700767 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:56.700853 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:56.700948 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:56.703026 (+ 2078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:56.703972 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:56.706530 (+ 2558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:56.707191 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:56.707310 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:56.713672 (+ 6362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:56.713680 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:56.713686 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:56.713730 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:56.717134 (+ 3404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:56.718219 (+ 1085us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:56.718225 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:56.718229 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:56.718361 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:56.718992 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:56.718997 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:56.719000 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:56.719373 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:56.719510 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:56.720587 (+ 1077us) server_negotiation.cc:299] Negotiation successful
0620 18:38:56.720881 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":220,"threads_started":1}
I20260620 18:38:56.722414 31686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:56.697430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:38311 (local address 127.23.115.1:44467)
0620 18:38:56.698282 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:56.698330 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:56.700461 (+ 2131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:56.701207 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:56.701215 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:56.701952 (+ 737us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:56.702804 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:56.702826 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:56.704128 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:56.704132 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:56.706394 (+ 2262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:56.706410 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:56.707455 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:56.707489 (+ 34us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:56.707770 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:56.708531 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:56.708591 (+ 60us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:56.713344 (+ 4753us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:56.717286 (+ 3942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:56.717294 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:56.717311 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:56.717991 (+ 680us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:56.718524 (+ 533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:56.718528 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:56.718530 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:56.718842 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:56.719584 (+ 742us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:56.719590 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:56.720088 (+ 498us) client_negotiation.cc:769] Sending connection context
0620 18:38:56.720433 (+ 345us) client_negotiation.cc:240] Negotiation successful
0620 18:38:56.720972 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":362,"threads_started":1}
I20260620 18:38:56.742041 31685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:56.728256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.1:46793 (local address 127.23.115.62:38311)
0620 18:38:56.728450 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:56.728456 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:56.728955 (+ 499us) server_negotiation.cc:407] Connection header received
0620 18:38:56.729109 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:56.729115 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:56.729202 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:56.729315 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:56.730221 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:56.731358 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:56.733415 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:56.734084 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:56.734222 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:56.736369 (+ 2147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:56.736377 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:56.736381 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:56.736420 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:56.739065 (+ 2645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:56.739989 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:56.739994 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:56.739998 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:56.740152 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:56.740624 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:56.740629 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:56.740632 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:56.740916 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:56.741027 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:56.741610 (+ 583us) server_negotiation.cc:299] Negotiation successful
0620 18:38:56.741805 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 18:38:56.742355 31752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:56.728261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:38311 (local address 127.23.115.1:46793)
0620 18:38:56.728863 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:56.728881 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:56.729009 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:56.729348 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:56.729351 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:56.729806 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:56.730095 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:56.730104 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:56.731544 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:56.731550 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:56.733242 (+ 1692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:56.733271 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:56.734292 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:56.734297 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:56.734412 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:56.735080 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:56.735104 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:56.736151 (+ 1047us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:56.739275 (+ 3124us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:56.739279 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:56.739282 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:56.739878 (+ 596us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:56.740285 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:56.740291 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:56.740295 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:56.740487 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:56.741029 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:56.741035 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:56.741252 (+ 217us) client_negotiation.cc:769] Sending connection context
0620 18:38:56.741476 (+ 224us) client_negotiation.cc:240] Negotiation successful
0620 18:38:56.741841 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":305,"threads_started":1}
I20260620 18:38:56.750684 31661 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.1:35375
I20260620 18:38:56.750710 31796 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.1:35375 every 8 connection(s)
I20260620 18:38:56.752288 31661 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980676253068-24012-0/minicluster-data/ts-0/data/info.pb
I20260620 18:38:56.753104 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 31661
I20260620 18:38:56.753307 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980676253068-24012-0/minicluster-data/ts-0/wal/instance
I20260620 18:38:56.754079 24012 external_mini_cluster.cc:1428] Setting key 566466350494f7486a539297106db92c
I20260620 18:38:56.754464 31797 heartbeater.cc:347] Connected to a master server at 127.23.115.62:38311
I20260620 18:38:56.754669 31797 heartbeater.cc:464] Registering TS with master...
I20260620 18:38:56.755448 31797 heartbeater.cc:511] Master 127.23.115.62:38311 requested a full tablet report, sending...
I20260620 18:38:56.757441 31590 ts_manager.cc:194] Registered new tserver with Master: 2dac9332ea0345ceb33c40b1779316b2 (127.23.115.1:35375)
I20260620 18:38:56.758849 31590 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.115.1@KRBTEST.COM'} at 127.23.115.1:44467
WARNING: no policy specified for oryx/127.23.115.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.115.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.115.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980676253068-24012-0/krb5kdc/oryx_127.23.115.2.keytab.
Entry for principal oryx/127.23.115.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980676253068-24012-0/krb5kdc/oryx_127.23.115.2.keytab.
I20260620 18:38:56.797964 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980676253068-24012-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980676253068-24012-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980676253068-24012-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980676253068-24012-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.115.2:0
--local_ip_for_outbound_sockets=127.23.115.2
--webserver_interface=127.23.115.2
--webserver_port=0
--tserver_master_addrs=127.23.115.62:38311
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980676253068-24012-0/krb5kdc/oryx_127.23.115.2.keytab
--principal=oryx/127.23.115.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:45287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:38:57.101869 31801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:57.102141 31801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:57.102237 31801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:57.109934 31801 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:57.110054 31801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:57.110183 31801 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 18:38:57.110283 31801 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.115.2
I20260620 18:38:57.120227 31801 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:45287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980676253068-24012-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980676253068-24012-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980676253068-24012-0/krb5kdc/oryx_127.23.115.2.keytab
--principal=oryx/127.23.115.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.115.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980676253068-24012-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.115.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.115.62:38311
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.115.2
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980676253068-24012-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:38:57.121454 31801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:57.122674 31801 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:57.135012 31807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:57.135180 31806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:57.135198 31809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:57.136336 31801 server_base.cc:1034] running on GCE node
I20260620 18:38:57.136925 31801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:57.137807 31801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:57.139084 31801 hybrid_clock.cc:648] HybridClock initialized: now 1781980737139014 us; error 43 us; skew 500 ppm
I20260620 18:38:57.139351 31801 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:38:57 dist-test-slave-d645 krb5kdc[31545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980737, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.115.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:57.142269 31801 init.cc:377] Logged in from keytab as oryx/127.23.115.2@KRBTEST.COM (short username oryx)
I20260620 18:38:57.143721 31801 webserver.cc:460] Webserver started at http://127.23.115.2:45389/ using document root <none> and password file <none>
I20260620 18:38:57.144351 31801 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:57.144428 31801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:57.144686 31801 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:38:57.147504 31801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980676253068-24012-0/minicluster-data/ts-1/data/instance:
uuid: "011792c5d0a64360a012de15ace8f1c7"
format_stamp: "Formatted at 2026-06-20 18:38:57 on dist-test-slave-d645"
server_key: "ebbc150265e296be60e62b0d751da1a8"
server_key_iv: "d8bc15bcd543ba3007e5b2ab23881760"
server_key_version: "encryptionkey@0"
I20260620 18:38:57.148219 31801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980676253068-24012-0/minicluster-data/ts-1/wal/instance:
uuid: "011792c5d0a64360a012de15ace8f1c7"
format_stamp: "Formatted at 2026-06-20 18:38:57 on dist-test-slave-d645"
server_key: "ebbc150265e296be60e62b0d751da1a8"
server_key_iv: "d8bc15bcd543ba3007e5b2ab23881760"
server_key_version: "encryptionkey@0"
I20260620 18:38:57.152956 31801 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 18:38:57.156370 31816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:57.157689 31801 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:38:57.157828 31801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980676253068-24012-0/minicluster-data/ts-1/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980676253068-24012-0/minicluster-data/ts-1/wal
uuid: "011792c5d0a64360a012de15ace8f1c7"
format_stamp: "Formatted at 2026-06-20 18:38:57 on dist-test-slave-d645"
server_key: "ebbc150265e296be60e62b0d751da1a8"
server_key_iv: "d8bc15bcd543ba3007e5b2ab23881760"
server_key_version: "encryptionkey@0"
I20260620 18:38:57.158016 31801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980676253068-24012-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980676253068-24012-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980676253068-24012-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:57.177062 31801 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:57.181628 31801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:57.181882 31801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:57.184759 31801 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:38:57.184855 31801 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:57.184995 31801 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:38:57.185076 31801 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 18:38:57 dist-test-slave-d645 krb5kdc[31545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980737, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.115.2@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM
I20260620 18:38:57.207213 31685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:57.185100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.2:47209 (local address 127.23.115.62:38311)
0620 18:38:57.185305 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:57.185311 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:57.186597 (+ 1286us) server_negotiation.cc:407] Connection header received
0620 18:38:57.188647 (+ 2050us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:57.188654 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:57.188735 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:57.188822 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:57.190661 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:57.191607 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:57.194313 (+ 2706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:57.195075 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:57.195238 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:57.199701 (+ 4463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:57.199706 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:57.199710 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:57.199744 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:57.202822 (+ 3078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:57.203923 (+ 1101us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:57.203929 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:57.203933 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:57.204112 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:57.204727 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:57.204732 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:57.204735 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:57.205117 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:57.205266 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:57.206622 (+ 1356us) server_negotiation.cc:299] Negotiation successful
0620 18:38:57.206843 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 18:38:57.208276 31825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:57.185586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:38311 (local address 127.23.115.2:47209)
0620 18:38:57.186410 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:57.186453 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:57.188365 (+ 1912us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:57.189041 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:57.189049 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:57.189683 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:57.190483 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:57.190507 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:57.191756 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:57.191761 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:57.194150 (+ 2389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:57.194168 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:57.195344 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:57.195353 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:57.195623 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:57.196438 (+ 815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:57.196469 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:57.199461 (+ 2992us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:57.203092 (+ 3631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:57.203102 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:57.203120 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:57.203753 (+ 633us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:57.204287 (+ 534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:57.204292 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:57.204295 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:57.204576 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:57.205384 (+ 808us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:57.205396 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:57.205932 (+ 536us) client_negotiation.cc:769] Sending connection context
0620 18:38:57.206231 (+ 299us) client_negotiation.cc:240] Negotiation successful
0620 18:38:57.206857 (+ 626us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":337,"threads_started":1}
I20260620 18:38:57.231549 31685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:57.214294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.2:53221 (local address 127.23.115.62:38311)
0620 18:38:57.214520 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:57.214527 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:57.219524 (+ 4997us) server_negotiation.cc:407] Connection header received
0620 18:38:57.219662 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:57.219669 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:57.219763 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:57.219891 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:57.220822 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:57.221557 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:57.222822 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:57.223308 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:57.223419 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:57.225757 (+ 2338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:57.225764 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:57.225769 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:57.225810 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:57.228870 (+ 3060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:57.229659 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:57.229666 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:57.229670 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:57.229800 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:57.230129 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:57.230135 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:57.230138 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:57.230470 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:57.230604 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:57.231020 (+ 416us) server_negotiation.cc:299] Negotiation successful
0620 18:38:57.231219 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 18:38:57.231649 31893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:57.214202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:38311 (local address 127.23.115.2:53221)
0620 18:38:57.219391 (+ 5189us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:57.219412 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:57.219551 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:57.219903 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:57.219906 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:57.220325 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:57.220673 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:57.220684 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:57.221689 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:57.221694 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:57.222689 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:57.222701 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:57.223505 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:57.223515 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:57.223713 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:57.224377 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:57.224403 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:57.225558 (+ 1155us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:57.229013 (+ 3455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:57.229017 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:57.229019 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:57.229551 (+ 532us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:57.229905 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:57.229910 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:57.229914 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:57.230033 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:57.230581 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:57.230586 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:57.230757 (+ 171us) client_negotiation.cc:769] Sending connection context
0620 18:38:57.230879 (+ 122us) client_negotiation.cc:240] Negotiation successful
0620 18:38:57.231211 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5034,"thread_start_us":4911,"threads_started":1}
I20260620 18:38:57.236284 31801 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.2:32777
I20260620 18:38:57.236310 31935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.2:32777 every 8 connection(s)
I20260620 18:38:57.237385 31801 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980676253068-24012-0/minicluster-data/ts-1/data/info.pb
I20260620 18:38:57.239282 31936 heartbeater.cc:347] Connected to a master server at 127.23.115.62:38311
I20260620 18:38:57.239465 31936 heartbeater.cc:464] Registering TS with master...
I20260620 18:38:57.240047 31936 heartbeater.cc:511] Master 127.23.115.62:38311 requested a full tablet report, sending...
I20260620 18:38:57.241211 31590 ts_manager.cc:194] Registered new tserver with Master: 011792c5d0a64360a012de15ace8f1c7 (127.23.115.2:32777)
I20260620 18:38:57.242003 31590 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.115.2@KRBTEST.COM'} at 127.23.115.2:47209
I20260620 18:38:57.247303 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 31801
I20260620 18:38:57.247440 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980676253068-24012-0/minicluster-data/ts-1/wal/instance
I20260620 18:38:57.247819 24012 external_mini_cluster.cc:1428] Setting key c1963f284fc8bc944acc01275f378b82
WARNING: no policy specified for oryx/127.23.115.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.115.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.115.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980676253068-24012-0/krb5kdc/oryx_127.23.115.3.keytab.
Entry for principal oryx/127.23.115.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980676253068-24012-0/krb5kdc/oryx_127.23.115.3.keytab.
I20260620 18:38:57.295063 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980676253068-24012-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980676253068-24012-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980676253068-24012-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980676253068-24012-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.115.3:0
--local_ip_for_outbound_sockets=127.23.115.3
--webserver_interface=127.23.115.3
--webserver_port=0
--tserver_master_addrs=127.23.115.62:38311
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980676253068-24012-0/krb5kdc/oryx_127.23.115.3.keytab
--principal=oryx/127.23.115.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:45287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:38:57.610180 31940 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:38:57.610479 31940 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:38:57.610600 31940 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:38:57.619463 31940 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:38:57.619589 31940 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:38:57.619707 31940 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 18:38:57.619815 31940 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.115.3
I20260620 18:38:57.630020 31940 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:45287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980676253068-24012-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980676253068-24012-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980676253068-24012-0/krb5kdc/oryx_127.23.115.3.keytab
--principal=oryx/127.23.115.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.115.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980676253068-24012-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.115.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.115.62:38311
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.115.3
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980676253068-24012-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:38:57.631757 31940 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:38:57.633437 31940 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:38:57.645669 31948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:57.645715 31945 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:38:57.645686 31946 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:38:57.646879 31940 server_base.cc:1034] running on GCE node
I20260620 18:38:57.647504 31940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:38:57.648437 31940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:38:57.649662 31940 hybrid_clock.cc:648] HybridClock initialized: now 1781980737649598 us; error 37 us; skew 500 ppm
I20260620 18:38:57.649962 31940 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:38:57 dist-test-slave-d645 krb5kdc[31545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980737, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.115.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:38:57.653378 31940 init.cc:377] Logged in from keytab as oryx/127.23.115.3@KRBTEST.COM (short username oryx)
I20260620 18:38:57.654767 31940 webserver.cc:460] Webserver started at http://127.23.115.3:42521/ using document root <none> and password file <none>
I20260620 18:38:57.655556 31940 fs_manager.cc:359] Metadata directory not provided
I20260620 18:38:57.655655 31940 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:38:57.655972 31940 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:38:57.658720 31940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980676253068-24012-0/minicluster-data/ts-2/data/instance:
uuid: "526ad504f2c7496a8b22c190efe68692"
format_stamp: "Formatted at 2026-06-20 18:38:57 on dist-test-slave-d645"
server_key: "b8f22d5e1d94738cff825fc309abfd1b"
server_key_iv: "d9bbeb3e3c2f3f417c4e14f9c1dcfa09"
server_key_version: "encryptionkey@0"
I20260620 18:38:57.659543 31940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980676253068-24012-0/minicluster-data/ts-2/wal/instance:
uuid: "526ad504f2c7496a8b22c190efe68692"
format_stamp: "Formatted at 2026-06-20 18:38:57 on dist-test-slave-d645"
server_key: "b8f22d5e1d94738cff825fc309abfd1b"
server_key_iv: "d9bbeb3e3c2f3f417c4e14f9c1dcfa09"
server_key_version: "encryptionkey@0"
I20260620 18:38:57.664445 31940 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 18:38:57.667766 31955 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:57.669071 31940 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:38:57.669222 31940 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980676253068-24012-0/minicluster-data/ts-2/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980676253068-24012-0/minicluster-data/ts-2/wal
uuid: "526ad504f2c7496a8b22c190efe68692"
format_stamp: "Formatted at 2026-06-20 18:38:57 on dist-test-slave-d645"
server_key: "b8f22d5e1d94738cff825fc309abfd1b"
server_key_iv: "d9bbeb3e3c2f3f417c4e14f9c1dcfa09"
server_key_version: "encryptionkey@0"
I20260620 18:38:57.669422 31940 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980676253068-24012-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980676253068-24012-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980676253068-24012-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:38:57.695873 31940 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:38:57.700137 31940 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:38:57.700398 31940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:38:57.703347 31940 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:38:57.703444 31940 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:38:57.703585 31940 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:38:57.703706 31940 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
Jun 20 18:38:57 dist-test-slave-d645 krb5kdc[31545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980737, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.115.3@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM
I20260620 18:38:57.724654 31685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:57.703938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.3:50205 (local address 127.23.115.62:38311)
0620 18:38:57.704145 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:57.704151 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:57.705774 (+ 1623us) server_negotiation.cc:407] Connection header received
0620 18:38:57.707740 (+ 1966us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:57.707746 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:57.707831 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:57.707916 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:57.709423 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:57.710188 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:57.712334 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:57.712981 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:57.713114 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:57.717721 (+ 4607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:57.717730 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:57.717736 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:57.717782 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:57.721140 (+ 3358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:57.722082 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:57.722101 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:57.722106 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:57.722309 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:57.722750 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:57.722756 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:57.722759 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:57.723167 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:57.723299 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:57.724053 (+ 754us) server_negotiation.cc:299] Negotiation successful
0620 18:38:57.724282 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 18:38:57.725656 31964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:57.704658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:38311 (local address 127.23.115.3:50205)
0620 18:38:57.705560 (+ 902us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:57.705607 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:57.707490 (+ 1883us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:57.708144 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:57.708152 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:57.708612 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:57.709258 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:57.709274 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:57.710334 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:57.710338 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:57.712171 (+ 1833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:57.712188 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:57.713430 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:57.713438 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:57.713752 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:57.714616 (+ 864us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:57.714651 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:57.717465 (+ 2814us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:57.721281 (+ 3816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:57.721290 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:57.721307 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:57.721875 (+ 568us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:57.722430 (+ 555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:57.722433 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:57.722436 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:57.722621 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:57.723301 (+ 680us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:57.723310 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:57.723626 (+ 316us) client_negotiation.cc:769] Sending connection context
0620 18:38:57.723864 (+ 238us) client_negotiation.cc:240] Negotiation successful
0620 18:38:57.724416 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"thread_start_us":360,"threads_started":1}
I20260620 18:38:57.746876 31685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:57.731846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.3:37399 (local address 127.23.115.62:38311)
0620 18:38:57.732006 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:57.732010 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:57.732572 (+ 562us) server_negotiation.cc:407] Connection header received
0620 18:38:57.732714 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:57.732720 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:57.732809 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:57.732952 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:57.733880 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:57.735073 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:57.737346 (+ 2273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:57.738009 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:57.738134 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:57.740576 (+ 2442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:57.740584 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:57.740590 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:57.740630 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:57.743464 (+ 2834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:57.744400 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:57.744406 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:57.744410 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:57.744567 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:57.745041 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:57.745046 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:57.745049 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:57.745406 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:57.745546 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:57.746410 (+ 864us) server_negotiation.cc:299] Negotiation successful
0620 18:38:57.746626 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 18:38:57.747030 32008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:57.731473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:38311 (local address 127.23.115.3:37399)
0620 18:38:57.732472 (+ 999us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:57.732491 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:57.732610 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:57.733005 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:57.733009 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:57.733454 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:57.733735 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:57.733745 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:57.735243 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:57.735249 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:57.737137 (+ 1888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:57.737163 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:57.738200 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:57.738208 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:57.738372 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:57.739069 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:57.739095 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:57.740312 (+ 1217us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:57.743598 (+ 3286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:57.743604 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:57.743608 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:57.744230 (+ 622us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:57.744706 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:57.744711 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:57.744716 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:57.744897 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:57.745610 (+ 713us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:57.745615 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:57.745881 (+ 266us) client_negotiation.cc:769] Sending connection context
0620 18:38:57.746065 (+ 184us) client_negotiation.cc:240] Negotiation successful
0620 18:38:57.746478 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":859,"thread_start_us":705,"threads_started":1}
I20260620 18:38:57.756987 31940 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.3:34889
I20260620 18:38:57.757009 32074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.3:34889 every 8 connection(s)
I20260620 18:38:57.758133 31940 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980676253068-24012-0/minicluster-data/ts-2/data/info.pb
I20260620 18:38:57.760309 32075 heartbeater.cc:347] Connected to a master server at 127.23.115.62:38311
I20260620 18:38:57.760505 32075 heartbeater.cc:464] Registering TS with master...
I20260620 18:38:57.761132 32075 heartbeater.cc:511] Master 127.23.115.62:38311 requested a full tablet report, sending...
I20260620 18:38:57.761902 31797 heartbeater.cc:503] Master 127.23.115.62:38311 was elected leader, sending a full tablet report...
I20260620 18:38:57.762456 31590 ts_manager.cc:194] Registered new tserver with Master: 526ad504f2c7496a8b22c190efe68692 (127.23.115.3:34889)
I20260620 18:38:57.763553 31590 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.115.3@KRBTEST.COM'} at 127.23.115.3:50205
I20260620 18:38:57.766180 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 31940
I20260620 18:38:57.766328 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980676253068-24012-0/minicluster-data/ts-2/wal/instance
I20260620 18:38:57.766716 24012 external_mini_cluster.cc:1428] Setting key 92d8077437be59a6d5a875e92381d731
I20260620 18:38:57.769416 24012 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 18:38:57 dist-test-slave-d645 krb5kdc[31545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980737, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 18:38:57 dist-test-slave-d645 krb5kdc[31545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.23.115.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:38:57 dist-test-slave-d645 krb5kdc[31545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.23.115.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:38:58.244274 31936 heartbeater.cc:503] Master 127.23.115.62:38311 was elected leader, sending a full tablet report...
I20260620 18:38:58.765692 32075 heartbeater.cc:503] Master 127.23.115.62:38311 was elected leader, sending a full tablet report...
W20260620 18:38:58.787961 32083 sasl_common.cc:323] Time spent calling sasl_client_start(): real 0.978s user 0.000s sys 0.013s
W20260620 18:38:58.789187 31685 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:38:57.803992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49754 (local address 127.23.115.62:38311)
0620 18:38:57.804161 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:57.804165 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:57.804511 (+ 346us) server_negotiation.cc:407] Connection header received
0620 18:38:57.804665 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:57.804669 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:57.804767 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:57.804875 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:57.806227 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:57.806997 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:57.808245 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:57.808672 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:57.808770 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:58.788892 (+980122us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49754: BlockingRecv error: recv got EOF from 127.0.0.1:49754 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 20 18:38:58 dist-test-slave-d645 krb5kdc[31545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980737, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM
I20260620 18:38:58.809188 31685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:58.793230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49760 (local address 127.23.115.62:38311)
0620 18:38:58.793453 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:58.793460 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:58.793736 (+ 276us) server_negotiation.cc:407] Connection header received
0620 18:38:58.793902 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:58.793908 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:58.794000 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:58.794088 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:58.795387 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:58.796103 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:58.797324 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:58.797759 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:58.797869 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:58.804190 (+ 6321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:58.804197 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:58.804200 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:58.804238 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:58.806962 (+ 2724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:58.807643 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:58.807648 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:58.807650 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:58.807782 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:58.808072 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:58.808075 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:58.808077 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:58.808359 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:58.808568 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:58.808774 (+ 206us) server_negotiation.cc:299] Negotiation successful
0620 18:38:58.808954 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 18:38:58.813355 31590 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49760:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:38:58.820084 31590 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:38:58.846287 32095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:58.832109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44144 (local address 127.23.115.1:35375)
0620 18:38:58.838317 (+ 6208us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:58.838331 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:58.838354 (+ 23us) server_negotiation.cc:407] Connection header received
0620 18:38:58.838435 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:58.838441 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:58.838626 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:58.838786 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:38:58.839650 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:58.841100 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:58.843937 (+ 2837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:58.845275 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:58.845378 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:58.845533 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:58.845921 (+ 388us) server_negotiation.cc:299] Negotiation successful
0620 18:38:58.846072 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6062,"thread_start_us":5925,"threads_started":1}
I20260620 18:38:58.846390 32096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:58.831905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.1:35375 (local address 127.0.0.1:44144)
0620 18:38:58.837482 (+ 5577us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:58.837533 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:58.837740 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:58.838834 (+ 1094us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:58.838843 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:58.838872 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:38:58.839423 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:58.839437 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:58.841240 (+ 1803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:58.841245 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:58.843713 (+ 2468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:58.843728 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:58.845454 (+ 1726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:58.845461 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:58.845644 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:58.845712 (+ 68us) client_negotiation.cc:769] Sending connection context
0620 18:38:58.845845 (+ 133us) client_negotiation.cc:240] Negotiation successful
0620 18:38:58.846036 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5250,"thread_start_us":7200,"threads_started":1}
I20260620 18:38:58.849722 32094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:58.832126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.2:32777 (local address 127.0.0.1:41286)
0620 18:38:58.840427 (+ 8301us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:58.840446 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:58.840555 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:58.841602 (+ 1047us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:58.841607 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:58.841632 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:38:58.842040 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:58.842052 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:58.845047 (+ 2995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:58.845052 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:58.846874 (+ 1822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:58.846886 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:58.848734 (+ 1848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:58.848740 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:58.849088 (+ 348us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:58.849128 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 18:38:58.849216 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 18:38:58.849371 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7126,"thread_start_us":4858,"threads_started":1}
I20260620 18:38:58.850390 32097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:58.832599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41286 (local address 127.23.115.2:32777)
0620 18:38:58.841017 (+ 8418us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:58.841025 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:58.841052 (+ 27us) server_negotiation.cc:407] Connection header received
0620 18:38:58.841142 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:58.841149 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:58.841398 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:58.842231 (+ 833us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:38:58.842430 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:58.844917 (+ 2487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:58.847032 (+ 2115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:58.848205 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:58.848345 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:58.848574 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:58.849733 (+ 1159us) server_negotiation.cc:299] Negotiation successful
0620 18:38:58.850117 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8270,"thread_start_us":8153,"threads_started":1}
I20260620 18:38:58.850713 31727 tablet_service.cc:1467] Processing CreateTablet for tablet c315d2b82d374d528c2bd434cd769434 (DEFAULT_TABLE table=test-table [id=bbe6545ade87469689c41b4210835296]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:38:58.852205 31727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c315d2b82d374d528c2bd434cd769434. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:58.852875 32099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:58.839450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.3:34889 (local address 127.0.0.1:55944)
0620 18:38:58.843672 (+ 4222us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:58.843691 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:58.843821 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:58.844593 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:58.844598 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:58.844622 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:38:58.844902 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:58.844910 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:58.848293 (+ 3383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:58.848297 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:58.850328 (+ 2031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:58.850342 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:58.852094 (+ 1752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:58.852100 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:58.852321 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:58.852355 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 18:38:58.852450 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 18:38:58.852575 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4104,"thread_start_us":3954,"threads_started":1}
I20260620 18:38:58.852932 32098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:58.838261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55944 (local address 127.23.115.3:34889)
0620 18:38:58.843311 (+ 5050us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:58.843319 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:58.843938 (+ 619us) server_negotiation.cc:407] Connection header received
0620 18:38:58.844038 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:58.844045 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:58.844290 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:58.844466 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:38:58.846788 (+ 2322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:58.848159 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:58.850502 (+ 2343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:58.851885 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:58.852020 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:58.852242 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:58.852499 (+ 257us) server_negotiation.cc:299] Negotiation successful
0620 18:38:58.852658 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4890,"thread_start_us":4774,"threads_started":1}
I20260620 18:38:58.853813 31866 tablet_service.cc:1467] Processing CreateTablet for tablet c315d2b82d374d528c2bd434cd769434 (DEFAULT_TABLE table=test-table [id=bbe6545ade87469689c41b4210835296]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:38:58.855494 31866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c315d2b82d374d528c2bd434cd769434. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:58.856585 32010 tablet_service.cc:1467] Processing CreateTablet for tablet c315d2b82d374d528c2bd434cd769434 (DEFAULT_TABLE table=test-table [id=bbe6545ade87469689c41b4210835296]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:38:58.857905 32010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c315d2b82d374d528c2bd434cd769434. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:58.866597 32100 tablet_bootstrap.cc:492] T c315d2b82d374d528c2bd434cd769434 P 011792c5d0a64360a012de15ace8f1c7: Bootstrap starting.
I20260620 18:38:58.867499 32101 tablet_bootstrap.cc:492] T c315d2b82d374d528c2bd434cd769434 P 2dac9332ea0345ceb33c40b1779316b2: Bootstrap starting.
I20260620 18:38:58.870759 32102 tablet_bootstrap.cc:492] T c315d2b82d374d528c2bd434cd769434 P 526ad504f2c7496a8b22c190efe68692: Bootstrap starting.
I20260620 18:38:58.870952 32101 tablet_bootstrap.cc:654] T c315d2b82d374d528c2bd434cd769434 P 2dac9332ea0345ceb33c40b1779316b2: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:58.871955 32100 tablet_bootstrap.cc:654] T c315d2b82d374d528c2bd434cd769434 P 011792c5d0a64360a012de15ace8f1c7: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:58.872790 32101 log.cc:824] T c315d2b82d374d528c2bd434cd769434 P 2dac9332ea0345ceb33c40b1779316b2: Log is configured to *not* fsync() on all Append() calls
I20260620 18:38:58.873332 32100 log.cc:824] T c315d2b82d374d528c2bd434cd769434 P 011792c5d0a64360a012de15ace8f1c7: Log is configured to *not* fsync() on all Append() calls
I20260620 18:38:58.873898 32102 tablet_bootstrap.cc:654] T c315d2b82d374d528c2bd434cd769434 P 526ad504f2c7496a8b22c190efe68692: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:58.874773 32102 log.cc:824] T c315d2b82d374d528c2bd434cd769434 P 526ad504f2c7496a8b22c190efe68692: Log is configured to *not* fsync() on all Append() calls
I20260620 18:38:58.876055 32101 tablet_bootstrap.cc:492] T c315d2b82d374d528c2bd434cd769434 P 2dac9332ea0345ceb33c40b1779316b2: No bootstrap required, opened a new log
I20260620 18:38:58.876304 32101 ts_tablet_manager.cc:1397] T c315d2b82d374d528c2bd434cd769434 P 2dac9332ea0345ceb33c40b1779316b2: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.006s
I20260620 18:38:58.877233 32100 tablet_bootstrap.cc:492] T c315d2b82d374d528c2bd434cd769434 P 011792c5d0a64360a012de15ace8f1c7: No bootstrap required, opened a new log
I20260620 18:38:58.877490 32102 tablet_bootstrap.cc:492] T c315d2b82d374d528c2bd434cd769434 P 526ad504f2c7496a8b22c190efe68692: No bootstrap required, opened a new log
I20260620 18:38:58.877619 32100 ts_tablet_manager.cc:1397] T c315d2b82d374d528c2bd434cd769434 P 011792c5d0a64360a012de15ace8f1c7: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260620 18:38:58.877728 32102 ts_tablet_manager.cc:1397] T c315d2b82d374d528c2bd434cd769434 P 526ad504f2c7496a8b22c190efe68692: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260620 18:38:58.883816 32101 raft_consensus.cc:348] T c315d2b82d374d528c2bd434cd769434 P 2dac9332ea0345ceb33c40b1779316b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dac9332ea0345ceb33c40b1779316b2" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 35375 } } peers { permanent_uuid: "011792c5d0a64360a012de15ace8f1c7" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 32777 } } peers { permanent_uuid: "526ad504f2c7496a8b22c190efe68692" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 34889 } }
I20260620 18:38:58.884187 32101 raft_consensus.cc:374] T c315d2b82d374d528c2bd434cd769434 P 2dac9332ea0345ceb33c40b1779316b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:58.884269 32101 raft_consensus.cc:729] T c315d2b82d374d528c2bd434cd769434 P 2dac9332ea0345ceb33c40b1779316b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2dac9332ea0345ceb33c40b1779316b2, State: Initialized, Role: FOLLOWER
I20260620 18:38:58.884845 32101 consensus_queue.cc:260] T c315d2b82d374d528c2bd434cd769434 P 2dac9332ea0345ceb33c40b1779316b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dac9332ea0345ceb33c40b1779316b2" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 35375 } } peers { permanent_uuid: "011792c5d0a64360a012de15ace8f1c7" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 32777 } } peers { permanent_uuid: "526ad504f2c7496a8b22c190efe68692" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 34889 } }
I20260620 18:38:58.886309 32101 ts_tablet_manager.cc:1428] T c315d2b82d374d528c2bd434cd769434 P 2dac9332ea0345ceb33c40b1779316b2: Time spent starting tablet: real 0.010s user 0.009s sys 0.001s
I20260620 18:38:58.888140 32102 raft_consensus.cc:348] T c315d2b82d374d528c2bd434cd769434 P 526ad504f2c7496a8b22c190efe68692 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dac9332ea0345ceb33c40b1779316b2" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 35375 } } peers { permanent_uuid: "011792c5d0a64360a012de15ace8f1c7" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 32777 } } peers { permanent_uuid: "526ad504f2c7496a8b22c190efe68692" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 34889 } }
I20260620 18:38:58.888484 32102 raft_consensus.cc:374] T c315d2b82d374d528c2bd434cd769434 P 526ad504f2c7496a8b22c190efe68692 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:58.888577 32102 raft_consensus.cc:729] T c315d2b82d374d528c2bd434cd769434 P 526ad504f2c7496a8b22c190efe68692 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 526ad504f2c7496a8b22c190efe68692, State: Initialized, Role: FOLLOWER
I20260620 18:38:58.889119 32102 consensus_queue.cc:260] T c315d2b82d374d528c2bd434cd769434 P 526ad504f2c7496a8b22c190efe68692 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dac9332ea0345ceb33c40b1779316b2" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 35375 } } peers { permanent_uuid: "011792c5d0a64360a012de15ace8f1c7" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 32777 } } peers { permanent_uuid: "526ad504f2c7496a8b22c190efe68692" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 34889 } }
I20260620 18:38:58.889027 32100 raft_consensus.cc:348] T c315d2b82d374d528c2bd434cd769434 P 011792c5d0a64360a012de15ace8f1c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dac9332ea0345ceb33c40b1779316b2" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 35375 } } peers { permanent_uuid: "011792c5d0a64360a012de15ace8f1c7" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 32777 } } peers { permanent_uuid: "526ad504f2c7496a8b22c190efe68692" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 34889 } }
I20260620 18:38:58.889459 32100 raft_consensus.cc:374] T c315d2b82d374d528c2bd434cd769434 P 011792c5d0a64360a012de15ace8f1c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:58.889561 32100 raft_consensus.cc:729] T c315d2b82d374d528c2bd434cd769434 P 011792c5d0a64360a012de15ace8f1c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 011792c5d0a64360a012de15ace8f1c7, State: Initialized, Role: FOLLOWER
I20260620 18:38:58.890224 32100 consensus_queue.cc:260] T c315d2b82d374d528c2bd434cd769434 P 011792c5d0a64360a012de15ace8f1c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dac9332ea0345ceb33c40b1779316b2" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 35375 } } peers { permanent_uuid: "011792c5d0a64360a012de15ace8f1c7" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 32777 } } peers { permanent_uuid: "526ad504f2c7496a8b22c190efe68692" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 34889 } }
I20260620 18:38:58.890875 32102 ts_tablet_manager.cc:1428] T c315d2b82d374d528c2bd434cd769434 P 526ad504f2c7496a8b22c190efe68692: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260620 18:38:58.892872 32100 ts_tablet_manager.cc:1428] T c315d2b82d374d528c2bd434cd769434 P 011792c5d0a64360a012de15ace8f1c7: Time spent starting tablet: real 0.015s user 0.009s sys 0.005s
W20260620 18:38:58.990056 31937 tablet.cc:2369] T c315d2b82d374d528c2bd434cd769434 P 011792c5d0a64360a012de15ace8f1c7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:38:59.006179 31798 tablet.cc:2369] T c315d2b82d374d528c2bd434cd769434 P 2dac9332ea0345ceb33c40b1779316b2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:38:59.010426 32076 tablet.cc:2369] T c315d2b82d374d528c2bd434cd769434 P 526ad504f2c7496a8b22c190efe68692: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:38:59.202152 32108 raft_consensus.cc:482] T c315d2b82d374d528c2bd434cd769434 P 011792c5d0a64360a012de15ace8f1c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:38:59.202435 32108 raft_consensus.cc:504] T c315d2b82d374d528c2bd434cd769434 P 011792c5d0a64360a012de15ace8f1c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dac9332ea0345ceb33c40b1779316b2" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 35375 } } peers { permanent_uuid: "011792c5d0a64360a012de15ace8f1c7" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 32777 } } peers { permanent_uuid: "526ad504f2c7496a8b22c190efe68692" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 34889 } }
I20260620 18:38:59.204161 32108 leader_election.cc:288] T c315d2b82d374d528c2bd434cd769434 P 011792c5d0a64360a012de15ace8f1c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2dac9332ea0345ceb33c40b1779316b2 (127.23.115.1:35375), 526ad504f2c7496a8b22c190efe68692 (127.23.115.3:34889)
I20260620 18:38:59.212708 32095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:59.204648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.2:42003 (local address 127.23.115.1:35375)
0620 18:38:59.204853 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:59.204858 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:59.205384 (+ 526us) server_negotiation.cc:407] Connection header received
0620 18:38:59.205498 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:59.205502 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:59.205567 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:59.205659 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:38:59.206444 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:59.207558 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:59.210441 (+ 2883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:59.211717 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:59.211886 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:59.211963 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:59.212284 (+ 321us) server_negotiation.cc:299] Negotiation successful
0620 18:38:59.212452 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 18:38:59.213132 32098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:59.204586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.2:51485 (local address 127.23.115.3:34889)
0620 18:38:59.204838 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:59.204845 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:59.205272 (+ 427us) server_negotiation.cc:407] Connection header received
0620 18:38:59.205483 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:59.205488 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:59.205565 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:59.205660 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:38:59.206453 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:59.207551 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:59.211067 (+ 3516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:59.212117 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:59.212241 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:59.212305 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:59.212778 (+ 473us) server_negotiation.cc:299] Negotiation successful
0620 18:38:59.212916 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 18:38:59.213284 32109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:59.204454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.3:34889 (local address 127.23.115.2:51485)
0620 18:38:59.205159 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:59.205179 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:59.205340 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:59.205739 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:59.205745 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:59.205836 (+ 91us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:38:59.206310 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:59.206320 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:59.207705 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:59.207709 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:59.210857 (+ 3148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:59.210874 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:59.212473 (+ 1599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:59.212479 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:59.212635 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:59.212664 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 18:38:59.212752 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 18:38:59.212892 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"mutex_wait_us":717,"spinlock_wait_cycles":23296,"thread_start_us":386,"threads_started":1}
I20260620 18:38:59.213438 32110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:59.204524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.1:35375 (local address 127.23.115.2:42003)
0620 18:38:59.205279 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:59.205294 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:59.205403 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:59.205739 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:59.205745 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:59.205777 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:38:59.206243 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:59.206263 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:59.207703 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:59.207709 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:59.210275 (+ 2566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:59.210293 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:59.211931 (+ 1638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:59.211938 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:59.212103 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:59.212137 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 18:38:59.212966 (+ 829us) client_negotiation.cc:240] Negotiation successful
0620 18:38:59.213094 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"mutex_wait_us":147,"thread_start_us":476,"threads_started":1}
I20260620 18:38:59.214270 32030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c315d2b82d374d528c2bd434cd769434" candidate_uuid: "011792c5d0a64360a012de15ace8f1c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "526ad504f2c7496a8b22c190efe68692" is_pre_election: true
I20260620 18:38:59.214270 31751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c315d2b82d374d528c2bd434cd769434" candidate_uuid: "011792c5d0a64360a012de15ace8f1c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2dac9332ea0345ceb33c40b1779316b2" is_pre_election: true
I20260620 18:38:59.214722 32030 raft_consensus.cc:2454] T c315d2b82d374d528c2bd434cd769434 P 526ad504f2c7496a8b22c190efe68692 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 011792c5d0a64360a012de15ace8f1c7 in term 0.
I20260620 18:38:59.214722 31751 raft_consensus.cc:2454] T c315d2b82d374d528c2bd434cd769434 P 2dac9332ea0345ceb33c40b1779316b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 011792c5d0a64360a012de15ace8f1c7 in term 0.
I20260620 18:38:59.215338 31818 leader_election.cc:302] T c315d2b82d374d528c2bd434cd769434 P 011792c5d0a64360a012de15ace8f1c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 011792c5d0a64360a012de15ace8f1c7, 526ad504f2c7496a8b22c190efe68692; no voters:
I20260620 18:38:59.215709 32108 raft_consensus.cc:2781] T c315d2b82d374d528c2bd434cd769434 P 011792c5d0a64360a012de15ace8f1c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:38:59.215823 32108 raft_consensus.cc:482] T c315d2b82d374d528c2bd434cd769434 P 011792c5d0a64360a012de15ace8f1c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:38:59.215930 32108 raft_consensus.cc:3037] T c315d2b82d374d528c2bd434cd769434 P 011792c5d0a64360a012de15ace8f1c7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:59.218183 32108 raft_consensus.cc:504] T c315d2b82d374d528c2bd434cd769434 P 011792c5d0a64360a012de15ace8f1c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dac9332ea0345ceb33c40b1779316b2" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 35375 } } peers { permanent_uuid: "011792c5d0a64360a012de15ace8f1c7" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 32777 } } peers { permanent_uuid: "526ad504f2c7496a8b22c190efe68692" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 34889 } }
I20260620 18:38:59.218761 32108 leader_election.cc:288] T c315d2b82d374d528c2bd434cd769434 P 011792c5d0a64360a012de15ace8f1c7 [CANDIDATE]: Term 1 election: Requested vote from peers 2dac9332ea0345ceb33c40b1779316b2 (127.23.115.1:35375), 526ad504f2c7496a8b22c190efe68692 (127.23.115.3:34889)
I20260620 18:38:59.219195 31751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c315d2b82d374d528c2bd434cd769434" candidate_uuid: "011792c5d0a64360a012de15ace8f1c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2dac9332ea0345ceb33c40b1779316b2"
I20260620 18:38:59.219254 32030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c315d2b82d374d528c2bd434cd769434" candidate_uuid: "011792c5d0a64360a012de15ace8f1c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "526ad504f2c7496a8b22c190efe68692"
I20260620 18:38:59.219367 31751 raft_consensus.cc:3037] T c315d2b82d374d528c2bd434cd769434 P 2dac9332ea0345ceb33c40b1779316b2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:59.219463 32030 raft_consensus.cc:3037] T c315d2b82d374d528c2bd434cd769434 P 526ad504f2c7496a8b22c190efe68692 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:59.221401 31751 raft_consensus.cc:2454] T c315d2b82d374d528c2bd434cd769434 P 2dac9332ea0345ceb33c40b1779316b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 011792c5d0a64360a012de15ace8f1c7 in term 1.
I20260620 18:38:59.221539 32030 raft_consensus.cc:2454] T c315d2b82d374d528c2bd434cd769434 P 526ad504f2c7496a8b22c190efe68692 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 011792c5d0a64360a012de15ace8f1c7 in term 1.
I20260620 18:38:59.221755 31819 leader_election.cc:302] T c315d2b82d374d528c2bd434cd769434 P 011792c5d0a64360a012de15ace8f1c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 011792c5d0a64360a012de15ace8f1c7, 2dac9332ea0345ceb33c40b1779316b2; no voters:
I20260620 18:38:59.222047 32108 raft_consensus.cc:2781] T c315d2b82d374d528c2bd434cd769434 P 011792c5d0a64360a012de15ace8f1c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:38:59.222450 32108 raft_consensus.cc:686] T c315d2b82d374d528c2bd434cd769434 P 011792c5d0a64360a012de15ace8f1c7 [term 1 LEADER]: Becoming Leader. State: Replica: 011792c5d0a64360a012de15ace8f1c7, State: Running, Role: LEADER
I20260620 18:38:59.222978 32108 consensus_queue.cc:237] T c315d2b82d374d528c2bd434cd769434 P 011792c5d0a64360a012de15ace8f1c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dac9332ea0345ceb33c40b1779316b2" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 35375 } } peers { permanent_uuid: "011792c5d0a64360a012de15ace8f1c7" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 32777 } } peers { permanent_uuid: "526ad504f2c7496a8b22c190efe68692" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 34889 } }
I20260620 18:38:59.227787 31590 catalog_manager.cc:5495] T c315d2b82d374d528c2bd434cd769434 P 011792c5d0a64360a012de15ace8f1c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 011792c5d0a64360a012de15ace8f1c7 (127.23.115.2). New cstate: current_term: 1 leader_uuid: "011792c5d0a64360a012de15ace8f1c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dac9332ea0345ceb33c40b1779316b2" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 35375 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "011792c5d0a64360a012de15ace8f1c7" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 32777 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "526ad504f2c7496a8b22c190efe68692" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 34889 } health_report { overall_health: UNKNOWN } } }
I20260620 18:38:59.300508 32097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:59.294474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41296 (local address 127.23.115.2:32777)
0620 18:38:59.294687 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:59.294692 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:59.294711 (+ 19us) server_negotiation.cc:407] Connection header received
0620 18:38:59.294993 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:59.294998 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:59.295081 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:59.295211 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:38:59.295578 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:59.296441 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:59.297913 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:59.298347 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:59.298453 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:59.298923 (+ 470us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:38:59.299776 (+ 853us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:38:59.299906 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:59.300111 (+ 205us) server_negotiation.cc:299] Negotiation successful
0620 18:38:59.300202 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260620 18:38:59.308709 31751 raft_consensus.cc:1261] T c315d2b82d374d528c2bd434cd769434 P 2dac9332ea0345ceb33c40b1779316b2 [term 1 FOLLOWER]: Refusing update from remote peer 011792c5d0a64360a012de15ace8f1c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:38:59.308709 32030 raft_consensus.cc:1261] T c315d2b82d374d528c2bd434cd769434 P 526ad504f2c7496a8b22c190efe68692 [term 1 FOLLOWER]: Refusing update from remote peer 011792c5d0a64360a012de15ace8f1c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:38:59.309662 32111 consensus_queue.cc:1035] T c315d2b82d374d528c2bd434cd769434 P 011792c5d0a64360a012de15ace8f1c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2dac9332ea0345ceb33c40b1779316b2" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 35375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:38:59.310024 32108 consensus_queue.cc:1035] T c315d2b82d374d528c2bd434cd769434 P 011792c5d0a64360a012de15ace8f1c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "526ad504f2c7496a8b22c190efe68692" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 34889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:38:59.321033 32115 mvcc.cc:205] Tried to move back new op lower bound from 7298993108198805504 to 7298993107861422080. Current Snapshot: MvccSnapshot[applied={T|T < 7298993108198805504}]
I20260620 18:38:59.329843 32116 mvcc.cc:205] Tried to move back new op lower bound from 7298993108198805504 to 7298993107861422080. Current Snapshot: MvccSnapshot[applied={T|T < 7298993108198805504 or (T in {7298993108198805504})}]
I20260620 18:38:59.334566 31589 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49760:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:38:59.334854 31589 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49760:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:38:59.338112 31589 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 522d0826a3f64949a4b30e203c4b3a74: Sending DeleteTablet for 3 replicas of tablet c315d2b82d374d528c2bd434cd769434
I20260620 18:38:59.339154 31727 tablet_service.cc:1514] Processing DeleteTablet for tablet c315d2b82d374d528c2bd434cd769434 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:38:59 UTC) from {username='oryx', principal='oryx/127.23.115.62@KRBTEST.COM'} at 127.0.0.1:44144
I20260620 18:38:59.339337 32010 tablet_service.cc:1514] Processing DeleteTablet for tablet c315d2b82d374d528c2bd434cd769434 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:38:59 UTC) from {username='oryx', principal='oryx/127.23.115.62@KRBTEST.COM'} at 127.0.0.1:55944
I20260620 18:38:59.340178 31866 tablet_service.cc:1514] Processing DeleteTablet for tablet c315d2b82d374d528c2bd434cd769434 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:38:59 UTC) from {username='oryx', principal='oryx/127.23.115.62@KRBTEST.COM'} at 127.0.0.1:41286
I20260620 18:38:59.340979 32126 tablet_replica.cc:330] T c315d2b82d374d528c2bd434cd769434 P 011792c5d0a64360a012de15ace8f1c7: stopping tablet replica
I20260620 18:38:59.340807 31589 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49760:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:38:59.341431 31589 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:38:59.341636 32126 raft_consensus.cc:2229] T c315d2b82d374d528c2bd434cd769434 P 011792c5d0a64360a012de15ace8f1c7 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:38:59.342330 32126 raft_consensus.cc:2258] T c315d2b82d374d528c2bd434cd769434 P 011792c5d0a64360a012de15ace8f1c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:38:59.343451 32125 tablet_replica.cc:330] T c315d2b82d374d528c2bd434cd769434 P 526ad504f2c7496a8b22c190efe68692: stopping tablet replica
I20260620 18:38:59.344005 32125 raft_consensus.cc:2229] T c315d2b82d374d528c2bd434cd769434 P 526ad504f2c7496a8b22c190efe68692 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:38:59.344552 32126 ts_tablet_manager.cc:1905] T c315d2b82d374d528c2bd434cd769434 P 011792c5d0a64360a012de15ace8f1c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:38:59.344556 32125 raft_consensus.cc:2258] T c315d2b82d374d528c2bd434cd769434 P 526ad504f2c7496a8b22c190efe68692 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:38:59.347173 32125 ts_tablet_manager.cc:1905] T c315d2b82d374d528c2bd434cd769434 P 526ad504f2c7496a8b22c190efe68692: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:38:59.350696 31726 tablet_service.cc:1467] Processing CreateTablet for tablet 16c3d30d44d14451845b99309b997bf8 (DEFAULT_TABLE table=test-table [id=723d906015b04f3489df83e771ccdc65]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:38:59.351337 31726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16c3d30d44d14451845b99309b997bf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:59.351420 31864 tablet_service.cc:1467] Processing CreateTablet for tablet 16c3d30d44d14451845b99309b997bf8 (DEFAULT_TABLE table=test-table [id=723d906015b04f3489df83e771ccdc65]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:38:59.351926 31864 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16c3d30d44d14451845b99309b997bf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:59.354306 32125 ts_tablet_manager.cc:1918] T c315d2b82d374d528c2bd434cd769434 P 526ad504f2c7496a8b22c190efe68692: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:38:59.354501 32125 log.cc:1196] T c315d2b82d374d528c2bd434cd769434 P 526ad504f2c7496a8b22c190efe68692: Deleting WAL directory at /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980676253068-24012-0/minicluster-data/ts-2/wal/wals/c315d2b82d374d528c2bd434cd769434
I20260620 18:38:59.355121 32125 ts_tablet_manager.cc:1939] T c315d2b82d374d528c2bd434cd769434 P 526ad504f2c7496a8b22c190efe68692: Deleting consensus metadata
I20260620 18:38:59.356281 32010 tablet_service.cc:1467] Processing CreateTablet for tablet 16c3d30d44d14451845b99309b997bf8 (DEFAULT_TABLE table=test-table [id=723d906015b04f3489df83e771ccdc65]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:38:59.356691 32010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16c3d30d44d14451845b99309b997bf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:59.357106 31575 catalog_manager.cc:4841] TS 526ad504f2c7496a8b22c190efe68692 (127.23.115.3:34889): tablet c315d2b82d374d528c2bd434cd769434 (table test-table [id=bbe6545ade87469689c41b4210835296]) successfully deleted
I20260620 18:38:59.357740 32101 tablet_bootstrap.cc:492] T 16c3d30d44d14451845b99309b997bf8 P 2dac9332ea0345ceb33c40b1779316b2: Bootstrap starting.
I20260620 18:38:59.359328 32101 tablet_bootstrap.cc:654] T 16c3d30d44d14451845b99309b997bf8 P 2dac9332ea0345ceb33c40b1779316b2: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:59.361387 32126 ts_tablet_manager.cc:1918] T c315d2b82d374d528c2bd434cd769434 P 011792c5d0a64360a012de15ace8f1c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:38:59.361594 32126 log.cc:1196] T c315d2b82d374d528c2bd434cd769434 P 011792c5d0a64360a012de15ace8f1c7: Deleting WAL directory at /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980676253068-24012-0/minicluster-data/ts-1/wal/wals/c315d2b82d374d528c2bd434cd769434
I20260620 18:38:59.362120 32126 ts_tablet_manager.cc:1939] T c315d2b82d374d528c2bd434cd769434 P 011792c5d0a64360a012de15ace8f1c7: Deleting consensus metadata
I20260620 18:38:59.364308 31575 catalog_manager.cc:4841] TS 011792c5d0a64360a012de15ace8f1c7 (127.23.115.2:32777): tablet c315d2b82d374d528c2bd434cd769434 (table test-table [id=bbe6545ade87469689c41b4210835296]) successfully deleted
I20260620 18:38:59.371248 32101 tablet_bootstrap.cc:492] T 16c3d30d44d14451845b99309b997bf8 P 2dac9332ea0345ceb33c40b1779316b2: No bootstrap required, opened a new log
I20260620 18:38:59.371475 32101 ts_tablet_manager.cc:1397] T 16c3d30d44d14451845b99309b997bf8 P 2dac9332ea0345ceb33c40b1779316b2: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.004s
I20260620 18:38:59.372133 32100 tablet_bootstrap.cc:492] T 16c3d30d44d14451845b99309b997bf8 P 011792c5d0a64360a012de15ace8f1c7: Bootstrap starting.
I20260620 18:38:59.372470 32101 raft_consensus.cc:348] T 16c3d30d44d14451845b99309b997bf8 P 2dac9332ea0345ceb33c40b1779316b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dac9332ea0345ceb33c40b1779316b2" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 35375 } } peers { permanent_uuid: "011792c5d0a64360a012de15ace8f1c7" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 32777 } } peers { permanent_uuid: "526ad504f2c7496a8b22c190efe68692" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 34889 } }
I20260620 18:38:59.372898 32101 raft_consensus.cc:374] T 16c3d30d44d14451845b99309b997bf8 P 2dac9332ea0345ceb33c40b1779316b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:59.373097 32101 raft_consensus.cc:729] T 16c3d30d44d14451845b99309b997bf8 P 2dac9332ea0345ceb33c40b1779316b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2dac9332ea0345ceb33c40b1779316b2, State: Initialized, Role: FOLLOWER
I20260620 18:38:59.373469 32101 consensus_queue.cc:260] T 16c3d30d44d14451845b99309b997bf8 P 2dac9332ea0345ceb33c40b1779316b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dac9332ea0345ceb33c40b1779316b2" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 35375 } } peers { permanent_uuid: "011792c5d0a64360a012de15ace8f1c7" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 32777 } } peers { permanent_uuid: "526ad504f2c7496a8b22c190efe68692" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 34889 } }
I20260620 18:38:59.373899 32100 tablet_bootstrap.cc:654] T 16c3d30d44d14451845b99309b997bf8 P 011792c5d0a64360a012de15ace8f1c7: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:59.376680 32101 ts_tablet_manager.cc:1428] T 16c3d30d44d14451845b99309b997bf8 P 2dac9332ea0345ceb33c40b1779316b2: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260620 18:38:59.379189 32124 tablet_replica.cc:330] T c315d2b82d374d528c2bd434cd769434 P 2dac9332ea0345ceb33c40b1779316b2: stopping tablet replica
I20260620 18:38:59.379621 32124 raft_consensus.cc:2229] T c315d2b82d374d528c2bd434cd769434 P 2dac9332ea0345ceb33c40b1779316b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:38:59.379671 32102 tablet_bootstrap.cc:492] T 16c3d30d44d14451845b99309b997bf8 P 526ad504f2c7496a8b22c190efe68692: Bootstrap starting.
I20260620 18:38:59.380100 32124 raft_consensus.cc:2258] T c315d2b82d374d528c2bd434cd769434 P 2dac9332ea0345ceb33c40b1779316b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:38:59.380237 32100 tablet_bootstrap.cc:492] T 16c3d30d44d14451845b99309b997bf8 P 011792c5d0a64360a012de15ace8f1c7: No bootstrap required, opened a new log
I20260620 18:38:59.380411 32100 ts_tablet_manager.cc:1397] T 16c3d30d44d14451845b99309b997bf8 P 011792c5d0a64360a012de15ace8f1c7: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260620 18:38:59.381071 32100 raft_consensus.cc:348] T 16c3d30d44d14451845b99309b997bf8 P 011792c5d0a64360a012de15ace8f1c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dac9332ea0345ceb33c40b1779316b2" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 35375 } } peers { permanent_uuid: "011792c5d0a64360a012de15ace8f1c7" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 32777 } } peers { permanent_uuid: "526ad504f2c7496a8b22c190efe68692" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 34889 } }
I20260620 18:38:59.381223 32102 tablet_bootstrap.cc:654] T 16c3d30d44d14451845b99309b997bf8 P 526ad504f2c7496a8b22c190efe68692: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:59.381261 32100 raft_consensus.cc:374] T 16c3d30d44d14451845b99309b997bf8 P 011792c5d0a64360a012de15ace8f1c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:59.381368 32100 raft_consensus.cc:729] T 16c3d30d44d14451845b99309b997bf8 P 011792c5d0a64360a012de15ace8f1c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 011792c5d0a64360a012de15ace8f1c7, State: Initialized, Role: FOLLOWER
I20260620 18:38:59.381654 32100 consensus_queue.cc:260] T 16c3d30d44d14451845b99309b997bf8 P 011792c5d0a64360a012de15ace8f1c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dac9332ea0345ceb33c40b1779316b2" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 35375 } } peers { permanent_uuid: "011792c5d0a64360a012de15ace8f1c7" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 32777 } } peers { permanent_uuid: "526ad504f2c7496a8b22c190efe68692" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 34889 } }
I20260620 18:38:59.382220 32100 ts_tablet_manager.cc:1428] T 16c3d30d44d14451845b99309b997bf8 P 011792c5d0a64360a012de15ace8f1c7: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260620 18:38:59.383955 32102 tablet_bootstrap.cc:492] T 16c3d30d44d14451845b99309b997bf8 P 526ad504f2c7496a8b22c190efe68692: No bootstrap required, opened a new log
I20260620 18:38:59.384099 32102 ts_tablet_manager.cc:1397] T 16c3d30d44d14451845b99309b997bf8 P 526ad504f2c7496a8b22c190efe68692: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260620 18:38:59.384811 32102 raft_consensus.cc:348] T 16c3d30d44d14451845b99309b997bf8 P 526ad504f2c7496a8b22c190efe68692 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dac9332ea0345ceb33c40b1779316b2" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 35375 } } peers { permanent_uuid: "011792c5d0a64360a012de15ace8f1c7" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 32777 } } peers { permanent_uuid: "526ad504f2c7496a8b22c190efe68692" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 34889 } }
I20260620 18:38:59.385033 32102 raft_consensus.cc:374] T 16c3d30d44d14451845b99309b997bf8 P 526ad504f2c7496a8b22c190efe68692 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:59.385125 32102 raft_consensus.cc:729] T 16c3d30d44d14451845b99309b997bf8 P 526ad504f2c7496a8b22c190efe68692 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 526ad504f2c7496a8b22c190efe68692, State: Initialized, Role: FOLLOWER
I20260620 18:38:59.385372 32102 consensus_queue.cc:260] T 16c3d30d44d14451845b99309b997bf8 P 526ad504f2c7496a8b22c190efe68692 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dac9332ea0345ceb33c40b1779316b2" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 35375 } } peers { permanent_uuid: "011792c5d0a64360a012de15ace8f1c7" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 32777 } } peers { permanent_uuid: "526ad504f2c7496a8b22c190efe68692" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 34889 } }
I20260620 18:38:59.385931 32102 ts_tablet_manager.cc:1428] T 16c3d30d44d14451845b99309b997bf8 P 526ad504f2c7496a8b22c190efe68692: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260620 18:38:59.387123 32124 ts_tablet_manager.cc:1905] T c315d2b82d374d528c2bd434cd769434 P 2dac9332ea0345ceb33c40b1779316b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:38:59.394124 32124 ts_tablet_manager.cc:1918] T c315d2b82d374d528c2bd434cd769434 P 2dac9332ea0345ceb33c40b1779316b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:38:59.394292 32124 log.cc:1196] T c315d2b82d374d528c2bd434cd769434 P 2dac9332ea0345ceb33c40b1779316b2: Deleting WAL directory at /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980676253068-24012-0/minicluster-data/ts-0/wal/wals/c315d2b82d374d528c2bd434cd769434
I20260620 18:38:59.394805 32124 ts_tablet_manager.cc:1939] T c315d2b82d374d528c2bd434cd769434 P 2dac9332ea0345ceb33c40b1779316b2: Deleting consensus metadata
I20260620 18:38:59.396657 31576 catalog_manager.cc:4841] TS 2dac9332ea0345ceb33c40b1779316b2 (127.23.115.1:35375): tablet c315d2b82d374d528c2bd434cd769434 (table test-table [id=bbe6545ade87469689c41b4210835296]) successfully deleted
I20260620 18:38:59.462906 32107 raft_consensus.cc:482] T 16c3d30d44d14451845b99309b997bf8 P 526ad504f2c7496a8b22c190efe68692 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:38:59.463267 32107 raft_consensus.cc:504] T 16c3d30d44d14451845b99309b997bf8 P 526ad504f2c7496a8b22c190efe68692 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dac9332ea0345ceb33c40b1779316b2" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 35375 } } peers { permanent_uuid: "011792c5d0a64360a012de15ace8f1c7" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 32777 } } peers { permanent_uuid: "526ad504f2c7496a8b22c190efe68692" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 34889 } }
I20260620 18:38:59.465212 32107 leader_election.cc:288] T 16c3d30d44d14451845b99309b997bf8 P 526ad504f2c7496a8b22c190efe68692 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2dac9332ea0345ceb33c40b1779316b2 (127.23.115.1:35375), 011792c5d0a64360a012de15ace8f1c7 (127.23.115.2:32777)
I20260620 18:38:59.470980 32106 raft_consensus.cc:482] T 16c3d30d44d14451845b99309b997bf8 P 2dac9332ea0345ceb33c40b1779316b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:38:59.471220 32106 raft_consensus.cc:504] T 16c3d30d44d14451845b99309b997bf8 P 2dac9332ea0345ceb33c40b1779316b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dac9332ea0345ceb33c40b1779316b2" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 35375 } } peers { permanent_uuid: "011792c5d0a64360a012de15ace8f1c7" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 32777 } } peers { permanent_uuid: "526ad504f2c7496a8b22c190efe68692" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 34889 } }
I20260620 18:38:59.473613 32097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:59.465996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.3:49559 (local address 127.23.115.2:32777)
0620 18:38:59.466193 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:59.466199 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:59.466487 (+ 288us) server_negotiation.cc:407] Connection header received
0620 18:38:59.466831 (+ 344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:59.466836 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:59.466924 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:59.467040 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:38:59.467768 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:59.468610 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:59.470904 (+ 2294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:59.472135 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:59.472269 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:59.472356 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:59.473079 (+ 723us) server_negotiation.cc:299] Negotiation successful
0620 18:38:59.473222 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 18:38:59.473618 32106 leader_election.cc:288] T 16c3d30d44d14451845b99309b997bf8 P 2dac9332ea0345ceb33c40b1779316b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 011792c5d0a64360a012de15ace8f1c7 (127.23.115.2:32777), 526ad504f2c7496a8b22c190efe68692 (127.23.115.3:34889)
I20260620 18:38:59.474289 32095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:59.465642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.3:46581 (local address 127.23.115.1:35375)
0620 18:38:59.465825 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:59.465832 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:59.466922 (+ 1090us) server_negotiation.cc:407] Connection header received
0620 18:38:59.467129 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:59.467134 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:59.467203 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:59.467249 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:38:59.468091 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:59.469201 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:59.471662 (+ 2461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:59.473114 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:59.473242 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:59.473306 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:59.473865 (+ 559us) server_negotiation.cc:299] Negotiation successful
0620 18:38:59.474012 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 18:38:59.474370 32130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:59.465451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.2:32777 (local address 127.23.115.3:49559)
0620 18:38:59.466378 (+ 927us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:59.466398 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:59.466608 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:59.467090 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:59.467095 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:59.467122 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:38:59.467531 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:59.467544 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:59.468731 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:59.468735 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:59.470706 (+ 1971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:59.470722 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:59.472598 (+ 1876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:59.472604 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:59.472822 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:59.472921 (+ 99us) client_negotiation.cc:769] Sending connection context
0620 18:38:59.473874 (+ 953us) client_negotiation.cc:240] Negotiation successful
0620 18:38:59.474014 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":794,"mutex_wait_us":5,"thread_start_us":752,"threads_started":1}
I20260620 18:38:59.474987 32131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:59.465479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.1:35375 (local address 127.23.115.3:46581)
0620 18:38:59.466814 (+ 1335us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:59.466833 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:59.466968 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:59.467427 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:59.467430 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:59.467467 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:38:59.467934 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:59.467945 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:59.469360 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:59.469365 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:59.471461 (+ 2096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:59.471476 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:59.473506 (+ 2030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:59.473512 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:59.473688 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:59.473719 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 18:38:59.474486 (+ 767us) client_negotiation.cc:240] Negotiation successful
0620 18:38:59.474616 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1224,"mutex_wait_us":51,"thread_start_us":1065,"threads_started":1}
I20260620 18:38:59.475462 31886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16c3d30d44d14451845b99309b997bf8" candidate_uuid: "526ad504f2c7496a8b22c190efe68692" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "011792c5d0a64360a012de15ace8f1c7" is_pre_election: true
I20260620 18:38:59.475948 31886 raft_consensus.cc:2454] T 16c3d30d44d14451845b99309b997bf8 P 011792c5d0a64360a012de15ace8f1c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 526ad504f2c7496a8b22c190efe68692 in term 0.
I20260620 18:38:59.476729 31751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16c3d30d44d14451845b99309b997bf8" candidate_uuid: "526ad504f2c7496a8b22c190efe68692" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2dac9332ea0345ceb33c40b1779316b2" is_pre_election: true
I20260620 18:38:59.476727 31957 leader_election.cc:302] T 16c3d30d44d14451845b99309b997bf8 P 526ad504f2c7496a8b22c190efe68692 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 011792c5d0a64360a012de15ace8f1c7, 526ad504f2c7496a8b22c190efe68692; no voters:
I20260620 18:38:59.476965 31751 raft_consensus.cc:2454] T 16c3d30d44d14451845b99309b997bf8 P 2dac9332ea0345ceb33c40b1779316b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 526ad504f2c7496a8b22c190efe68692 in term 0.
I20260620 18:38:59.477196 32107 raft_consensus.cc:2781] T 16c3d30d44d14451845b99309b997bf8 P 526ad504f2c7496a8b22c190efe68692 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:38:59.477311 32107 raft_consensus.cc:482] T 16c3d30d44d14451845b99309b997bf8 P 526ad504f2c7496a8b22c190efe68692 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:38:59.477396 32107 raft_consensus.cc:3037] T 16c3d30d44d14451845b99309b997bf8 P 526ad504f2c7496a8b22c190efe68692 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:59.480711 32107 raft_consensus.cc:504] T 16c3d30d44d14451845b99309b997bf8 P 526ad504f2c7496a8b22c190efe68692 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dac9332ea0345ceb33c40b1779316b2" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 35375 } } peers { permanent_uuid: "011792c5d0a64360a012de15ace8f1c7" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 32777 } } peers { permanent_uuid: "526ad504f2c7496a8b22c190efe68692" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 34889 } }
I20260620 18:38:59.481899 31751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16c3d30d44d14451845b99309b997bf8" candidate_uuid: "526ad504f2c7496a8b22c190efe68692" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2dac9332ea0345ceb33c40b1779316b2"
I20260620 18:38:59.482126 31751 raft_consensus.cc:3037] T 16c3d30d44d14451845b99309b997bf8 P 2dac9332ea0345ceb33c40b1779316b2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:59.485471 31751 raft_consensus.cc:2454] T 16c3d30d44d14451845b99309b997bf8 P 2dac9332ea0345ceb33c40b1779316b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 526ad504f2c7496a8b22c190efe68692 in term 1.
I20260620 18:38:59.485772 32097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:59.474161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.1:44499 (local address 127.23.115.2:32777)
0620 18:38:59.474334 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:59.474339 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:59.477921 (+ 3582us) server_negotiation.cc:407] Connection header received
0620 18:38:59.477991 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:59.477996 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:59.478069 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:59.478151 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:38:59.478891 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:59.480046 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:59.483138 (+ 3092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:59.484428 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:59.484594 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:59.484693 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:59.485259 (+ 566us) server_negotiation.cc:299] Negotiation successful
0620 18:38:59.485413 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 18:38:59.485975 31958 leader_election.cc:302] T 16c3d30d44d14451845b99309b997bf8 P 526ad504f2c7496a8b22c190efe68692 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2dac9332ea0345ceb33c40b1779316b2, 526ad504f2c7496a8b22c190efe68692; no voters:
I20260620 18:38:59.486486 32132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:59.473868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.2:32777 (local address 127.23.115.1:44499)
0620 18:38:59.475678 (+ 1810us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:59.475696 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:59.475810 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:59.478302 (+ 2492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:59.478307 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:59.478340 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:38:59.478696 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:59.478708 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:59.480231 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:59.480517 (+ 286us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:59.482910 (+ 2393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:59.482927 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:59.484901 (+ 1974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:59.484907 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:59.485082 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:59.485114 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 18:38:59.486000 (+ 886us) client_negotiation.cc:240] Negotiation successful
0620 18:38:59.486170 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1683,"thread_start_us":1552,"threads_started":1}
I20260620 18:38:59.486618 32134 raft_consensus.cc:2781] T 16c3d30d44d14451845b99309b997bf8 P 526ad504f2c7496a8b22c190efe68692 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:38:59.487641 31886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16c3d30d44d14451845b99309b997bf8" candidate_uuid: "526ad504f2c7496a8b22c190efe68692" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "011792c5d0a64360a012de15ace8f1c7"
I20260620 18:38:59.487879 31886 raft_consensus.cc:3037] T 16c3d30d44d14451845b99309b997bf8 P 011792c5d0a64360a012de15ace8f1c7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:59.491122 31886 raft_consensus.cc:2454] T 16c3d30d44d14451845b99309b997bf8 P 011792c5d0a64360a012de15ace8f1c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 526ad504f2c7496a8b22c190efe68692 in term 1.
I20260620 18:38:59.491632 32107 leader_election.cc:288] T 16c3d30d44d14451845b99309b997bf8 P 526ad504f2c7496a8b22c190efe68692 [CANDIDATE]: Term 1 election: Requested vote from peers 2dac9332ea0345ceb33c40b1779316b2 (127.23.115.1:35375), 011792c5d0a64360a012de15ace8f1c7 (127.23.115.2:32777)
I20260620 18:38:59.492749 31885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16c3d30d44d14451845b99309b997bf8" candidate_uuid: "2dac9332ea0345ceb33c40b1779316b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "011792c5d0a64360a012de15ace8f1c7" is_pre_election: true
I20260620 18:38:59.493204 31885 raft_consensus.cc:2379] T 16c3d30d44d14451845b99309b997bf8 P 011792c5d0a64360a012de15ace8f1c7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2dac9332ea0345ceb33c40b1779316b2 in current term 1: Already voted for candidate 526ad504f2c7496a8b22c190efe68692 in this term.
I20260620 18:38:59.494561 32134 raft_consensus.cc:686] T 16c3d30d44d14451845b99309b997bf8 P 526ad504f2c7496a8b22c190efe68692 [term 1 LEADER]: Becoming Leader. State: Replica: 526ad504f2c7496a8b22c190efe68692, State: Running, Role: LEADER
I20260620 18:38:59.495078 32134 consensus_queue.cc:237] T 16c3d30d44d14451845b99309b997bf8 P 526ad504f2c7496a8b22c190efe68692 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dac9332ea0345ceb33c40b1779316b2" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 35375 } } peers { permanent_uuid: "011792c5d0a64360a012de15ace8f1c7" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 32777 } } peers { permanent_uuid: "526ad504f2c7496a8b22c190efe68692" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 34889 } }
I20260620 18:38:59.496349 32132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:59.475826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.3:34889 (local address 127.23.115.1:37991)
0620 18:38:59.486814 (+ 10988us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:59.486833 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:59.487018 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:59.487494 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:59.487499 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:59.487524 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:38:59.487690 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:59.487700 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:59.489183 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:59.489188 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:59.491182 (+ 1994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:59.491197 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:59.495585 (+ 4388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:59.495591 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:59.495782 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:59.495824 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 18:38:59.495919 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 18:38:59.496061 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10849,"thread_start_us":16197,"threads_started":1}
I20260620 18:38:59.496831 32098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:59.476073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.1:37991 (local address 127.23.115.3:34889)
0620 18:38:59.476252 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:59.476257 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:59.486899 (+ 10642us) server_negotiation.cc:407] Connection header received
0620 18:38:59.487162 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:59.487167 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:59.487244 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:59.487303 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:38:59.487873 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:59.488992 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:59.491380 (+ 2388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:59.495236 (+ 3856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:59.495414 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:59.495482 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:59.496001 (+ 519us) server_negotiation.cc:299] Negotiation successful
0620 18:38:59.496138 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 18:38:59.497500 32030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16c3d30d44d14451845b99309b997bf8" candidate_uuid: "2dac9332ea0345ceb33c40b1779316b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "526ad504f2c7496a8b22c190efe68692" is_pre_election: true
I20260620 18:38:59.498490 31678 leader_election.cc:302] T 16c3d30d44d14451845b99309b997bf8 P 2dac9332ea0345ceb33c40b1779316b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2dac9332ea0345ceb33c40b1779316b2; no voters: 011792c5d0a64360a012de15ace8f1c7, 526ad504f2c7496a8b22c190efe68692
I20260620 18:38:59.498878 32106 raft_consensus.cc:2726] T 16c3d30d44d14451845b99309b997bf8 P 2dac9332ea0345ceb33c40b1779316b2 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 18:38:59.499514 31590 catalog_manager.cc:5495] T 16c3d30d44d14451845b99309b997bf8 P 526ad504f2c7496a8b22c190efe68692 reported cstate change: term changed from 0 to 1, leader changed from <none> to 526ad504f2c7496a8b22c190efe68692 (127.23.115.3). New cstate: current_term: 1 leader_uuid: "526ad504f2c7496a8b22c190efe68692" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dac9332ea0345ceb33c40b1779316b2" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 35375 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "011792c5d0a64360a012de15ace8f1c7" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 32777 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "526ad504f2c7496a8b22c190efe68692" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 34889 } health_report { overall_health: HEALTHY } } }
Jun 20 18:38:59 dist-test-slave-d645 krb5kdc[31545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980736, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.115.62@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM
I20260620 18:38:59.550402 32142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:59.537385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:38311 (local address 127.0.0.1:49772)
0620 18:38:59.538014 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:59.538036 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:59.538209 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:59.538553 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:59.538558 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:59.539269 (+ 711us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:38:59.539457 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:59.539467 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:59.540494 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:59.540497 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:59.541392 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:59.541403 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:59.542077 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:59.542083 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:59.542259 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:59.542909 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:38:59.542960 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:38:59.545247 (+ 2287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:38:59.548016 (+ 2769us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:59.548028 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:59.548037 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:59.548427 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:59.548753 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:38:59.548758 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:38:59.548761 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:38:59.548898 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:38:59.549383 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:38:59.549391 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:38:59.549537 (+ 146us) client_negotiation.cc:769] Sending connection context
0620 18:38:59.549681 (+ 144us) client_negotiation.cc:240] Negotiation successful
0620 18:38:59.549946 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":336,"threads_started":1}
I20260620 18:38:59.550431 32143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:59.537413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49772 (local address 127.23.115.62:38311)
0620 18:38:59.538316 (+ 903us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:59.538321 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:59.538340 (+ 19us) server_negotiation.cc:407] Connection header received
0620 18:38:59.538377 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:59.538380 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:59.538431 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:59.538520 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:38:59.539605 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:59.540358 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:59.541514 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:59.541930 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:59.542034 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:59.545436 (+ 3402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:38:59.545441 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:38:59.545445 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:38:59.545473 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:38:59.547902 (+ 2429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:59.548537 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:59.548541 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:59.548543 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:59.548651 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:38:59.548994 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:38:59.548998 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:38:59.549001 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:38:59.549254 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:38:59.549397 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:59.549773 (+ 376us) server_negotiation.cc:299] Negotiation successful
0620 18:38:59.550069 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":121,"server-negotiator.queue_time_us":790,"thread_start_us":646,"threads_started":1}
I20260620 18:38:59.555989 31590 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.23.115.62@KRBTEST.COM'} at 127.0.0.1:49772:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 18:38:59.557049 31590 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:38:59.563510 31864 tablet_service.cc:1467] Processing CreateTablet for tablet f0b86e6cc90e4601ab452a5b664f4a84 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b34cbd8ef2aa40c1a13a9eaae13f4c42]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 18:38:59.563495 32010 tablet_service.cc:1467] Processing CreateTablet for tablet f0b86e6cc90e4601ab452a5b664f4a84 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b34cbd8ef2aa40c1a13a9eaae13f4c42]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 18:38:59.563823 31727 tablet_service.cc:1467] Processing CreateTablet for tablet f0b86e6cc90e4601ab452a5b664f4a84 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b34cbd8ef2aa40c1a13a9eaae13f4c42]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 18:38:59.564033 31864 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0b86e6cc90e4601ab452a5b664f4a84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:59.564026 32010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0b86e6cc90e4601ab452a5b664f4a84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:59.564309 31727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0b86e6cc90e4601ab452a5b664f4a84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:38:59.571034 32100 tablet_bootstrap.cc:492] T f0b86e6cc90e4601ab452a5b664f4a84 P 011792c5d0a64360a012de15ace8f1c7: Bootstrap starting.
I20260620 18:38:59.571179 32101 tablet_bootstrap.cc:492] T f0b86e6cc90e4601ab452a5b664f4a84 P 2dac9332ea0345ceb33c40b1779316b2: Bootstrap starting.
I20260620 18:38:59.572002 32102 tablet_bootstrap.cc:492] T f0b86e6cc90e4601ab452a5b664f4a84 P 526ad504f2c7496a8b22c190efe68692: Bootstrap starting.
I20260620 18:38:59.572796 32101 tablet_bootstrap.cc:654] T f0b86e6cc90e4601ab452a5b664f4a84 P 2dac9332ea0345ceb33c40b1779316b2: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:59.572832 32100 tablet_bootstrap.cc:654] T f0b86e6cc90e4601ab452a5b664f4a84 P 011792c5d0a64360a012de15ace8f1c7: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:59.573827 32102 tablet_bootstrap.cc:654] T f0b86e6cc90e4601ab452a5b664f4a84 P 526ad504f2c7496a8b22c190efe68692: Neither blocks nor log segments found. Creating new log.
I20260620 18:38:59.584256 32101 tablet_bootstrap.cc:492] T f0b86e6cc90e4601ab452a5b664f4a84 P 2dac9332ea0345ceb33c40b1779316b2: No bootstrap required, opened a new log
I20260620 18:38:59.584257 32102 tablet_bootstrap.cc:492] T f0b86e6cc90e4601ab452a5b664f4a84 P 526ad504f2c7496a8b22c190efe68692: No bootstrap required, opened a new log
I20260620 18:38:59.584439 32102 ts_tablet_manager.cc:1397] T f0b86e6cc90e4601ab452a5b664f4a84 P 526ad504f2c7496a8b22c190efe68692: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.001s
I20260620 18:38:59.584442 32101 ts_tablet_manager.cc:1397] T f0b86e6cc90e4601ab452a5b664f4a84 P 2dac9332ea0345ceb33c40b1779316b2: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.001s
I20260620 18:38:59.585014 32100 tablet_bootstrap.cc:492] T f0b86e6cc90e4601ab452a5b664f4a84 P 011792c5d0a64360a012de15ace8f1c7: No bootstrap required, opened a new log
I20260620 18:38:59.585273 32100 ts_tablet_manager.cc:1397] T f0b86e6cc90e4601ab452a5b664f4a84 P 011792c5d0a64360a012de15ace8f1c7: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.009s
I20260620 18:38:59.585155 32101 raft_consensus.cc:348] T f0b86e6cc90e4601ab452a5b664f4a84 P 2dac9332ea0345ceb33c40b1779316b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "011792c5d0a64360a012de15ace8f1c7" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 32777 } } peers { permanent_uuid: "526ad504f2c7496a8b22c190efe68692" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 34889 } } peers { permanent_uuid: "2dac9332ea0345ceb33c40b1779316b2" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 35375 } }
I20260620 18:38:59.585155 32102 raft_consensus.cc:348] T f0b86e6cc90e4601ab452a5b664f4a84 P 526ad504f2c7496a8b22c190efe68692 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "011792c5d0a64360a012de15ace8f1c7" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 32777 } } peers { permanent_uuid: "526ad504f2c7496a8b22c190efe68692" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 34889 } } peers { permanent_uuid: "2dac9332ea0345ceb33c40b1779316b2" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 35375 } }
I20260620 18:38:59.585384 32102 raft_consensus.cc:374] T f0b86e6cc90e4601ab452a5b664f4a84 P 526ad504f2c7496a8b22c190efe68692 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:59.585397 32101 raft_consensus.cc:374] T f0b86e6cc90e4601ab452a5b664f4a84 P 2dac9332ea0345ceb33c40b1779316b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:59.585469 32101 raft_consensus.cc:729] T f0b86e6cc90e4601ab452a5b664f4a84 P 2dac9332ea0345ceb33c40b1779316b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2dac9332ea0345ceb33c40b1779316b2, State: Initialized, Role: FOLLOWER
I20260620 18:38:59.585469 32102 raft_consensus.cc:729] T f0b86e6cc90e4601ab452a5b664f4a84 P 526ad504f2c7496a8b22c190efe68692 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 526ad504f2c7496a8b22c190efe68692, State: Initialized, Role: FOLLOWER
I20260620 18:38:59.585732 32101 consensus_queue.cc:260] T f0b86e6cc90e4601ab452a5b664f4a84 P 2dac9332ea0345ceb33c40b1779316b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "011792c5d0a64360a012de15ace8f1c7" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 32777 } } peers { permanent_uuid: "526ad504f2c7496a8b22c190efe68692" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 34889 } } peers { permanent_uuid: "2dac9332ea0345ceb33c40b1779316b2" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 35375 } }
I20260620 18:38:59.585734 32102 consensus_queue.cc:260] T f0b86e6cc90e4601ab452a5b664f4a84 P 526ad504f2c7496a8b22c190efe68692 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "011792c5d0a64360a012de15ace8f1c7" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 32777 } } peers { permanent_uuid: "526ad504f2c7496a8b22c190efe68692" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 34889 } } peers { permanent_uuid: "2dac9332ea0345ceb33c40b1779316b2" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 35375 } }
I20260620 18:38:59.586135 32134 tablet_replica.cc:439] T f0b86e6cc90e4601ab452a5b664f4a84 P 526ad504f2c7496a8b22c190efe68692: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "011792c5d0a64360a012de15ace8f1c7" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 32777 } } peers { permanent_uuid: "526ad504f2c7496a8b22c190efe68692" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 34889 } } peers { permanent_uuid: "2dac9332ea0345ceb33c40b1779316b2" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 35375 } } }
I20260620 18:38:59.586165 32106 tablet_replica.cc:439] T f0b86e6cc90e4601ab452a5b664f4a84 P 2dac9332ea0345ceb33c40b1779316b2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "011792c5d0a64360a012de15ace8f1c7" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 32777 } } peers { permanent_uuid: "526ad504f2c7496a8b22c190efe68692" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 34889 } } peers { permanent_uuid: "2dac9332ea0345ceb33c40b1779316b2" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 35375 } } }
I20260620 18:38:59.586409 32106 tablet_replica.cc:442] T f0b86e6cc90e4601ab452a5b664f4a84 P 2dac9332ea0345ceb33c40b1779316b2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:38:59.586427 32134 tablet_replica.cc:442] T f0b86e6cc90e4601ab452a5b664f4a84 P 526ad504f2c7496a8b22c190efe68692: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:38:59.586862 32101 ts_tablet_manager.cc:1428] T f0b86e6cc90e4601ab452a5b664f4a84 P 2dac9332ea0345ceb33c40b1779316b2: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260620 18:38:59.587560 32102 ts_tablet_manager.cc:1428] T f0b86e6cc90e4601ab452a5b664f4a84 P 526ad504f2c7496a8b22c190efe68692: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260620 18:38:59.587543 32100 raft_consensus.cc:348] T f0b86e6cc90e4601ab452a5b664f4a84 P 011792c5d0a64360a012de15ace8f1c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "011792c5d0a64360a012de15ace8f1c7" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 32777 } } peers { permanent_uuid: "526ad504f2c7496a8b22c190efe68692" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 34889 } } peers { permanent_uuid: "2dac9332ea0345ceb33c40b1779316b2" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 35375 } }
I20260620 18:38:59.587899 32100 raft_consensus.cc:374] T f0b86e6cc90e4601ab452a5b664f4a84 P 011792c5d0a64360a012de15ace8f1c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:38:59.588027 32100 raft_consensus.cc:729] T f0b86e6cc90e4601ab452a5b664f4a84 P 011792c5d0a64360a012de15ace8f1c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 011792c5d0a64360a012de15ace8f1c7, State: Initialized, Role: FOLLOWER
I20260620 18:38:59.588341 32100 consensus_queue.cc:260] T f0b86e6cc90e4601ab452a5b664f4a84 P 011792c5d0a64360a012de15ace8f1c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "011792c5d0a64360a012de15ace8f1c7" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 32777 } } peers { permanent_uuid: "526ad504f2c7496a8b22c190efe68692" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 34889 } } peers { permanent_uuid: "2dac9332ea0345ceb33c40b1779316b2" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 35375 } }
I20260620 18:38:59.589159 32108 tablet_replica.cc:439] T f0b86e6cc90e4601ab452a5b664f4a84 P 011792c5d0a64360a012de15ace8f1c7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "011792c5d0a64360a012de15ace8f1c7" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 32777 } } peers { permanent_uuid: "526ad504f2c7496a8b22c190efe68692" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 34889 } } peers { permanent_uuid: "2dac9332ea0345ceb33c40b1779316b2" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 35375 } } }
I20260620 18:38:59.589434 32108 tablet_replica.cc:442] T f0b86e6cc90e4601ab452a5b664f4a84 P 011792c5d0a64360a012de15ace8f1c7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:38:59.589970 32100 ts_tablet_manager.cc:1428] T f0b86e6cc90e4601ab452a5b664f4a84 P 011792c5d0a64360a012de15ace8f1c7: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260620 18:38:59.758122 32108 raft_consensus.cc:482] T f0b86e6cc90e4601ab452a5b664f4a84 P 011792c5d0a64360a012de15ace8f1c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:38:59.758383 32108 raft_consensus.cc:504] T f0b86e6cc90e4601ab452a5b664f4a84 P 011792c5d0a64360a012de15ace8f1c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "011792c5d0a64360a012de15ace8f1c7" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 32777 } } peers { permanent_uuid: "526ad504f2c7496a8b22c190efe68692" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 34889 } } peers { permanent_uuid: "2dac9332ea0345ceb33c40b1779316b2" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 35375 } }
I20260620 18:38:59.759258 32108 leader_election.cc:288] T f0b86e6cc90e4601ab452a5b664f4a84 P 011792c5d0a64360a012de15ace8f1c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 526ad504f2c7496a8b22c190efe68692 (127.23.115.3:34889), 2dac9332ea0345ceb33c40b1779316b2 (127.23.115.1:35375)
I20260620 18:38:59.760049 31751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0b86e6cc90e4601ab452a5b664f4a84" candidate_uuid: "011792c5d0a64360a012de15ace8f1c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2dac9332ea0345ceb33c40b1779316b2" is_pre_election: true
I20260620 18:38:59.760334 31751 raft_consensus.cc:2454] T f0b86e6cc90e4601ab452a5b664f4a84 P 2dac9332ea0345ceb33c40b1779316b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 011792c5d0a64360a012de15ace8f1c7 in term 0.
I20260620 18:38:59.760865 31819 leader_election.cc:302] T f0b86e6cc90e4601ab452a5b664f4a84 P 011792c5d0a64360a012de15ace8f1c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 011792c5d0a64360a012de15ace8f1c7, 2dac9332ea0345ceb33c40b1779316b2; no voters:
I20260620 18:38:59.761708 32030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0b86e6cc90e4601ab452a5b664f4a84" candidate_uuid: "011792c5d0a64360a012de15ace8f1c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "526ad504f2c7496a8b22c190efe68692" is_pre_election: true
I20260620 18:38:59.761960 32030 raft_consensus.cc:2454] T f0b86e6cc90e4601ab452a5b664f4a84 P 526ad504f2c7496a8b22c190efe68692 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 011792c5d0a64360a012de15ace8f1c7 in term 0.
I20260620 18:38:59.762619 32108 raft_consensus.cc:2781] T f0b86e6cc90e4601ab452a5b664f4a84 P 011792c5d0a64360a012de15ace8f1c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:38:59.762737 32108 raft_consensus.cc:482] T f0b86e6cc90e4601ab452a5b664f4a84 P 011792c5d0a64360a012de15ace8f1c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:38:59.762826 32108 raft_consensus.cc:3037] T f0b86e6cc90e4601ab452a5b664f4a84 P 011792c5d0a64360a012de15ace8f1c7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:59.765899 32108 raft_consensus.cc:504] T f0b86e6cc90e4601ab452a5b664f4a84 P 011792c5d0a64360a012de15ace8f1c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "011792c5d0a64360a012de15ace8f1c7" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 32777 } } peers { permanent_uuid: "526ad504f2c7496a8b22c190efe68692" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 34889 } } peers { permanent_uuid: "2dac9332ea0345ceb33c40b1779316b2" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 35375 } }
I20260620 18:38:59.766918 32030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0b86e6cc90e4601ab452a5b664f4a84" candidate_uuid: "011792c5d0a64360a012de15ace8f1c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "526ad504f2c7496a8b22c190efe68692"
I20260620 18:38:59.767179 32030 raft_consensus.cc:3037] T f0b86e6cc90e4601ab452a5b664f4a84 P 526ad504f2c7496a8b22c190efe68692 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:59.770140 32030 raft_consensus.cc:2454] T f0b86e6cc90e4601ab452a5b664f4a84 P 526ad504f2c7496a8b22c190efe68692 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 011792c5d0a64360a012de15ace8f1c7 in term 1.
I20260620 18:38:59.770675 31818 leader_election.cc:302] T f0b86e6cc90e4601ab452a5b664f4a84 P 011792c5d0a64360a012de15ace8f1c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 011792c5d0a64360a012de15ace8f1c7, 526ad504f2c7496a8b22c190efe68692; no voters:
I20260620 18:38:59.771032 32121 raft_consensus.cc:2781] T f0b86e6cc90e4601ab452a5b664f4a84 P 011792c5d0a64360a012de15ace8f1c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:38:59.771198 32121 raft_consensus.cc:686] T f0b86e6cc90e4601ab452a5b664f4a84 P 011792c5d0a64360a012de15ace8f1c7 [term 1 LEADER]: Becoming Leader. State: Replica: 011792c5d0a64360a012de15ace8f1c7, State: Running, Role: LEADER
I20260620 18:38:59.771538 32121 consensus_queue.cc:237] T f0b86e6cc90e4601ab452a5b664f4a84 P 011792c5d0a64360a012de15ace8f1c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "011792c5d0a64360a012de15ace8f1c7" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 32777 } } peers { permanent_uuid: "526ad504f2c7496a8b22c190efe68692" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 34889 } } peers { permanent_uuid: "2dac9332ea0345ceb33c40b1779316b2" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 35375 } }
I20260620 18:38:59.774660 31751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0b86e6cc90e4601ab452a5b664f4a84" candidate_uuid: "011792c5d0a64360a012de15ace8f1c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2dac9332ea0345ceb33c40b1779316b2"
I20260620 18:38:59.774888 31751 raft_consensus.cc:3037] T f0b86e6cc90e4601ab452a5b664f4a84 P 2dac9332ea0345ceb33c40b1779316b2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:38:59.777899 31751 raft_consensus.cc:2454] T f0b86e6cc90e4601ab452a5b664f4a84 P 2dac9332ea0345ceb33c40b1779316b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 011792c5d0a64360a012de15ace8f1c7 in term 1.
I20260620 18:38:59.779111 32111 tablet_replica.cc:439] T f0b86e6cc90e4601ab452a5b664f4a84 P 011792c5d0a64360a012de15ace8f1c7: TxnStatusTablet state changed. Reason: New leader 011792c5d0a64360a012de15ace8f1c7. Latest consensus state: current_term: 1 leader_uuid: "011792c5d0a64360a012de15ace8f1c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "011792c5d0a64360a012de15ace8f1c7" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 32777 } } peers { permanent_uuid: "526ad504f2c7496a8b22c190efe68692" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 34889 } } peers { permanent_uuid: "2dac9332ea0345ceb33c40b1779316b2" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 35375 } } }
I20260620 18:38:59.779405 32111 tablet_replica.cc:442] T f0b86e6cc90e4601ab452a5b664f4a84 P 011792c5d0a64360a012de15ace8f1c7: This TxnStatusTablet replica's current role is: LEADER
I20260620 18:38:59.781306 31590 catalog_manager.cc:5495] T f0b86e6cc90e4601ab452a5b664f4a84 P 011792c5d0a64360a012de15ace8f1c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 011792c5d0a64360a012de15ace8f1c7 (127.23.115.2). New cstate: current_term: 1 leader_uuid: "011792c5d0a64360a012de15ace8f1c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "011792c5d0a64360a012de15ace8f1c7" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 32777 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "526ad504f2c7496a8b22c190efe68692" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 34889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2dac9332ea0345ceb33c40b1779316b2" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 35375 } health_report { overall_health: UNKNOWN } } }
I20260620 18:38:59.781791 32108 leader_election.cc:288] T f0b86e6cc90e4601ab452a5b664f4a84 P 011792c5d0a64360a012de15ace8f1c7 [CANDIDATE]: Term 1 election: Requested vote from peers 526ad504f2c7496a8b22c190efe68692 (127.23.115.3:34889), 2dac9332ea0345ceb33c40b1779316b2 (127.23.115.1:35375)
I20260620 18:38:59.786108 32148 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 18:38:59.786345 32148 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 18:38:59.843576 32142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:59.837937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.2:32777 (local address 127.0.0.1:41306)
0620 18:38:59.838209 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:59.838228 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:59.838359 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:59.838810 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:59.838814 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:59.838825 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 18:38:59.839030 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:59.839039 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:59.840389 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:59.840393 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:59.841404 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:59.841416 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:59.842409 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:59.842413 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:59.842579 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:59.842636 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 18:38:59.843106 (+ 470us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 18:38:59.843133 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 18:38:59.843204 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 18:38:59.843346 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260620 18:38:59.843642 32097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:59.838097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41306 (local address 127.23.115.2:32777)
0620 18:38:59.838284 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:59.838291 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:59.838311 (+ 20us) server_negotiation.cc:407] Connection header received
0620 18:38:59.838534 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:59.838540 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:59.838627 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:59.838723 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:38:59.839169 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:59.840231 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:59.841539 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:59.842244 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:59.842387 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:59.842811 (+ 424us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:38:59.842989 (+ 178us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:38:59.843076 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:59.843242 (+ 166us) server_negotiation.cc:299] Negotiation successful
0620 18:38:59.843323 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 18:38:59.847352 31751 raft_consensus.cc:1261] T f0b86e6cc90e4601ab452a5b664f4a84 P 2dac9332ea0345ceb33c40b1779316b2 [term 1 FOLLOWER]: Refusing update from remote peer 011792c5d0a64360a012de15ace8f1c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:38:59.847390 32030 raft_consensus.cc:1261] T f0b86e6cc90e4601ab452a5b664f4a84 P 526ad504f2c7496a8b22c190efe68692 [term 1 FOLLOWER]: Refusing update from remote peer 011792c5d0a64360a012de15ace8f1c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:38:59.847929 32108 consensus_queue.cc:1035] T f0b86e6cc90e4601ab452a5b664f4a84 P 011792c5d0a64360a012de15ace8f1c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "526ad504f2c7496a8b22c190efe68692" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 34889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:38:59.848167 32111 consensus_queue.cc:1035] T f0b86e6cc90e4601ab452a5b664f4a84 P 011792c5d0a64360a012de15ace8f1c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2dac9332ea0345ceb33c40b1779316b2" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 35375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:38:59.856582 32134 tablet_replica.cc:439] T f0b86e6cc90e4601ab452a5b664f4a84 P 526ad504f2c7496a8b22c190efe68692: TxnStatusTablet state changed. Reason: New leader 011792c5d0a64360a012de15ace8f1c7. Latest consensus state: current_term: 1 leader_uuid: "011792c5d0a64360a012de15ace8f1c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "011792c5d0a64360a012de15ace8f1c7" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 32777 } } peers { permanent_uuid: "526ad504f2c7496a8b22c190efe68692" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 34889 } } peers { permanent_uuid: "2dac9332ea0345ceb33c40b1779316b2" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 35375 } } }
I20260620 18:38:59.856920 32134 tablet_replica.cc:442] T f0b86e6cc90e4601ab452a5b664f4a84 P 526ad504f2c7496a8b22c190efe68692: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:38:59.857863 32106 tablet_replica.cc:439] T f0b86e6cc90e4601ab452a5b664f4a84 P 2dac9332ea0345ceb33c40b1779316b2: TxnStatusTablet state changed. Reason: New leader 011792c5d0a64360a012de15ace8f1c7. Latest consensus state: current_term: 1 leader_uuid: "011792c5d0a64360a012de15ace8f1c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "011792c5d0a64360a012de15ace8f1c7" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 32777 } } peers { permanent_uuid: "526ad504f2c7496a8b22c190efe68692" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 34889 } } peers { permanent_uuid: "2dac9332ea0345ceb33c40b1779316b2" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 35375 } } }
I20260620 18:38:59.858166 32106 tablet_replica.cc:442] T f0b86e6cc90e4601ab452a5b664f4a84 P 2dac9332ea0345ceb33c40b1779316b2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:38:59.860440 32150 mvcc.cc:205] Tried to move back new op lower bound from 7298993110409019392 to 7298993110107205632. Current Snapshot: MvccSnapshot[applied={T|T < 7298993110409019392}]
I20260620 18:38:59.865235 32106 tablet_replica.cc:439] T f0b86e6cc90e4601ab452a5b664f4a84 P 2dac9332ea0345ceb33c40b1779316b2: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "011792c5d0a64360a012de15ace8f1c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "011792c5d0a64360a012de15ace8f1c7" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 32777 } } peers { permanent_uuid: "526ad504f2c7496a8b22c190efe68692" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 34889 } } peers { permanent_uuid: "2dac9332ea0345ceb33c40b1779316b2" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 35375 } } }
I20260620 18:38:59.865491 32106 tablet_replica.cc:442] T f0b86e6cc90e4601ab452a5b664f4a84 P 2dac9332ea0345ceb33c40b1779316b2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:38:59.867609 32134 tablet_replica.cc:439] T f0b86e6cc90e4601ab452a5b664f4a84 P 526ad504f2c7496a8b22c190efe68692: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "011792c5d0a64360a012de15ace8f1c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "011792c5d0a64360a012de15ace8f1c7" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 32777 } } peers { permanent_uuid: "526ad504f2c7496a8b22c190efe68692" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 34889 } } peers { permanent_uuid: "2dac9332ea0345ceb33c40b1779316b2" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 35375 } } }
I20260620 18:38:59.867836 32134 tablet_replica.cc:442] T f0b86e6cc90e4601ab452a5b664f4a84 P 526ad504f2c7496a8b22c190efe68692: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:38:59.869189 32111 tablet_replica.cc:439] T f0b86e6cc90e4601ab452a5b664f4a84 P 011792c5d0a64360a012de15ace8f1c7: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "011792c5d0a64360a012de15ace8f1c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "011792c5d0a64360a012de15ace8f1c7" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 32777 } } peers { permanent_uuid: "526ad504f2c7496a8b22c190efe68692" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 34889 } } peers { permanent_uuid: "2dac9332ea0345ceb33c40b1779316b2" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 35375 } } }
I20260620 18:38:59.869436 32111 tablet_replica.cc:442] T f0b86e6cc90e4601ab452a5b664f4a84 P 011792c5d0a64360a012de15ace8f1c7: This TxnStatusTablet replica's current role is: LEADER
I20260620 18:38:59.869822 32108 tablet_replica.cc:439] T f0b86e6cc90e4601ab452a5b664f4a84 P 011792c5d0a64360a012de15ace8f1c7: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "011792c5d0a64360a012de15ace8f1c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "011792c5d0a64360a012de15ace8f1c7" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 32777 } } peers { permanent_uuid: "526ad504f2c7496a8b22c190efe68692" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 34889 } } peers { permanent_uuid: "2dac9332ea0345ceb33c40b1779316b2" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 35375 } } }
I20260620 18:38:59.870114 32108 tablet_replica.cc:442] T f0b86e6cc90e4601ab452a5b664f4a84 P 011792c5d0a64360a012de15ace8f1c7: This TxnStatusTablet replica's current role is: LEADER
I20260620 18:38:59.881091 32098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:59.874314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55956 (local address 127.23.115.3:34889)
0620 18:38:59.874483 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:59.874489 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:59.875004 (+ 515us) server_negotiation.cc:407] Connection header received
0620 18:38:59.875225 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:59.875228 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:59.875324 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:59.875436 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:38:59.875886 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:59.876706 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:59.878667 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:59.879201 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:59.879336 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:59.879842 (+ 506us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:38:59.880414 (+ 572us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:38:59.880521 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:59.880712 (+ 191us) server_negotiation.cc:299] Negotiation successful
0620 18:38:59.880788 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 18:38:59.885775 32134 consensus_queue.cc:1035] T 16c3d30d44d14451845b99309b997bf8 P 526ad504f2c7496a8b22c190efe68692 [LEADER]: Connected to new peer: Peer: permanent_uuid: "011792c5d0a64360a012de15ace8f1c7" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 32777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:38:59.891822 32107 consensus_queue.cc:1035] T 16c3d30d44d14451845b99309b997bf8 P 526ad504f2c7496a8b22c190efe68692 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2dac9332ea0345ceb33c40b1779316b2" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 35375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:38:59.902060 32097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:59.892755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.3:43419 (local address 127.23.115.2:32777)
0620 18:38:59.892967 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:59.892974 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:59.896128 (+ 3154us) server_negotiation.cc:407] Connection header received
0620 18:38:59.896186 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:59.896190 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:59.896263 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:59.896347 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:38:59.897389 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:59.898498 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:59.900025 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:59.900679 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:59.900800 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:59.901185 (+ 385us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:38:59.901344 (+ 159us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:38:59.901433 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:59.901592 (+ 159us) server_negotiation.cc:299] Negotiation successful
0620 18:38:59.901668 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 18:38:59.902140 32164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:59.893219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.2:32777 (local address 127.23.115.3:43419)
0620 18:38:59.895144 (+ 1925us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:59.895163 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:59.895300 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:59.896844 (+ 1544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:59.896849 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:59.896862 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 18:38:59.897214 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:59.897226 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:59.898644 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:59.898649 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:59.899890 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:59.899903 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:59.900850 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:59.900855 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:59.900974 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:59.901026 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 18:38:59.901463 (+ 437us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 18:38:59.901474 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 18:38:59.901560 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 18:38:59.901682 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1762,"thread_start_us":4133,"threads_started":1}
I20260620 18:38:59.938839 32098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:59.932586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.2:41099 (local address 127.23.115.3:34889)
0620 18:38:59.932764 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 18:38:59.932771 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:38:59.933358 (+ 587us) server_negotiation.cc:407] Connection header received
0620 18:38:59.933586 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:38:59.933591 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:38:59.933683 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:38:59.933746 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:38:59.934495 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:59.935403 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:59.936849 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:38:59.937419 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:38:59.937547 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:59.938071 (+ 524us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:38:59.938207 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:38:59.938298 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 18:38:59.938436 (+ 138us) server_negotiation.cc:299] Negotiation successful
0620 18:38:59.938516 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 18:38:59.938988 32166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:38:59.932406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.3:34889 (local address 127.23.115.2:41099)
0620 18:38:59.933264 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0620 18:38:59.933288 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 18:38:59.933420 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:38:59.933931 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:38:59.933935 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:38:59.933947 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 18:38:59.934350 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:59.934360 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:59.935525 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:59.935531 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:59.936721 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:38:59.936733 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:38:59.937598 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:38:59.937605 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:38:59.937818 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:38:59.937893 (+ 75us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 18:38:59.938308 (+ 415us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 18:38:59.938320 (+ 12us) client_negotiation.cc:769] Sending connection context
0620 18:38:59.938404 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 18:38:59.938547 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":658,"thread_start_us":468,"threads_started":1}
I20260620 18:38:59.987308 31590 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49760:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:38:59.987584 31590 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49760:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:38:59.991348 31590 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 522d0826a3f64949a4b30e203c4b3a74: Sending DeleteTablet for 3 replicas of tablet 16c3d30d44d14451845b99309b997bf8
I20260620 18:38:59.992576 31727 tablet_service.cc:1514] Processing DeleteTablet for tablet 16c3d30d44d14451845b99309b997bf8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:38:59 UTC) from {username='oryx', principal='oryx/127.23.115.62@KRBTEST.COM'} at 127.0.0.1:44144
I20260620 18:38:59.993111 32169 tablet_replica.cc:330] T 16c3d30d44d14451845b99309b997bf8 P 2dac9332ea0345ceb33c40b1779316b2: stopping tablet replica
I20260620 18:38:59.993374 32169 raft_consensus.cc:2229] T 16c3d30d44d14451845b99309b997bf8 P 2dac9332ea0345ceb33c40b1779316b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:38:59.993655 32169 raft_consensus.cc:2258] T 16c3d30d44d14451845b99309b997bf8 P 2dac9332ea0345ceb33c40b1779316b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:38:59.993685 32010 tablet_service.cc:1514] Processing DeleteTablet for tablet 16c3d30d44d14451845b99309b997bf8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:38:59 UTC) from {username='oryx', principal='oryx/127.23.115.62@KRBTEST.COM'} at 127.0.0.1:55944
I20260620 18:38:59.994714 32169 ts_tablet_manager.cc:1905] T 16c3d30d44d14451845b99309b997bf8 P 2dac9332ea0345ceb33c40b1779316b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:38:59.995548 31864 tablet_service.cc:1514] Processing DeleteTablet for tablet 16c3d30d44d14451845b99309b997bf8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:38:59 UTC) from {username='oryx', principal='oryx/127.23.115.62@KRBTEST.COM'} at 127.0.0.1:41286
I20260620 18:39:00.002245 32169 ts_tablet_manager.cc:1918] T 16c3d30d44d14451845b99309b997bf8 P 2dac9332ea0345ceb33c40b1779316b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:39:00.002458 32169 log.cc:1196] T 16c3d30d44d14451845b99309b997bf8 P 2dac9332ea0345ceb33c40b1779316b2: Deleting WAL directory at /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980676253068-24012-0/minicluster-data/ts-0/wal/wals/16c3d30d44d14451845b99309b997bf8
I20260620 18:39:00.003041 32169 ts_tablet_manager.cc:1939] T 16c3d30d44d14451845b99309b997bf8 P 2dac9332ea0345ceb33c40b1779316b2: Deleting consensus metadata
I20260620 18:39:00.005692 31576 catalog_manager.cc:4841] TS 2dac9332ea0345ceb33c40b1779316b2 (127.23.115.1:35375): tablet 16c3d30d44d14451845b99309b997bf8 (table test-table [id=723d906015b04f3489df83e771ccdc65]) successfully deleted
I20260620 18:39:00.022085 32171 tablet_replica.cc:330] T 16c3d30d44d14451845b99309b997bf8 P 011792c5d0a64360a012de15ace8f1c7: stopping tablet replica
I20260620 18:39:00.022920 32170 tablet_replica.cc:330] T 16c3d30d44d14451845b99309b997bf8 P 526ad504f2c7496a8b22c190efe68692: stopping tablet replica
I20260620 18:39:00.023484 32170 raft_consensus.cc:2229] T 16c3d30d44d14451845b99309b997bf8 P 526ad504f2c7496a8b22c190efe68692 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:39:00.023840 32171 raft_consensus.cc:2229] T 16c3d30d44d14451845b99309b997bf8 P 011792c5d0a64360a012de15ace8f1c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:39:00.024226 32171 raft_consensus.cc:2258] T 16c3d30d44d14451845b99309b997bf8 P 011792c5d0a64360a012de15ace8f1c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:39:00.024905 32170 raft_consensus.cc:2258] T 16c3d30d44d14451845b99309b997bf8 P 526ad504f2c7496a8b22c190efe68692 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:39:00.026404 32170 ts_tablet_manager.cc:1905] T 16c3d30d44d14451845b99309b997bf8 P 526ad504f2c7496a8b22c190efe68692: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:39:00.026402 32171 ts_tablet_manager.cc:1905] T 16c3d30d44d14451845b99309b997bf8 P 011792c5d0a64360a012de15ace8f1c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:39:00.033556 32171 ts_tablet_manager.cc:1918] T 16c3d30d44d14451845b99309b997bf8 P 011792c5d0a64360a012de15ace8f1c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:39:00.033556 32170 ts_tablet_manager.cc:1918] T 16c3d30d44d14451845b99309b997bf8 P 526ad504f2c7496a8b22c190efe68692: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:39:00.033699 32171 log.cc:1196] T 16c3d30d44d14451845b99309b997bf8 P 011792c5d0a64360a012de15ace8f1c7: Deleting WAL directory at /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980676253068-24012-0/minicluster-data/ts-1/wal/wals/16c3d30d44d14451845b99309b997bf8
I20260620 18:39:00.033828 32170 log.cc:1196] T 16c3d30d44d14451845b99309b997bf8 P 526ad504f2c7496a8b22c190efe68692: Deleting WAL directory at /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980676253068-24012-0/minicluster-data/ts-2/wal/wals/16c3d30d44d14451845b99309b997bf8
I20260620 18:39:00.034159 32171 ts_tablet_manager.cc:1939] T 16c3d30d44d14451845b99309b997bf8 P 011792c5d0a64360a012de15ace8f1c7: Deleting consensus metadata
I20260620 18:39:00.034359 32170 ts_tablet_manager.cc:1939] T 16c3d30d44d14451845b99309b997bf8 P 526ad504f2c7496a8b22c190efe68692: Deleting consensus metadata
I20260620 18:39:00.035459 31575 catalog_manager.cc:4841] TS 526ad504f2c7496a8b22c190efe68692 (127.23.115.3:34889): tablet 16c3d30d44d14451845b99309b997bf8 (table test-table [id=723d906015b04f3489df83e771ccdc65]) successfully deleted
I20260620 18:39:00.039296 31575 catalog_manager.cc:4841] TS 011792c5d0a64360a012de15ace8f1c7 (127.23.115.2:32777): tablet 16c3d30d44d14451845b99309b997bf8 (table test-table [id=723d906015b04f3489df83e771ccdc65]) successfully deleted
Jun 20 18:39:00 dist-test-slave-d645 krb5kdc[31545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980740, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 18:39:00 dist-test-slave-d645 krb5kdc[31545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.23.115.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:39:00 dist-test-slave-d645 krb5kdc[31545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.23.115.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 18:39:00.096424 32179 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:00.063057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49776 (local address 127.23.115.62:38311)
0620 18:39:00.079284 (+ 16227us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:00.079291 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:00.079315 (+ 24us) server_negotiation.cc:407] Connection header received
0620 18:39:00.079380 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:00.079385 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:00.079465 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:00.079578 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:00.081364 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:00.082487 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:00.084335 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:00.084943 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:00.085078 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:00.096118 (+ 11040us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49776: BlockingRecv error: recv got EOF from 127.0.0.1:49776 (error 108)
Metrics: {"server-negotiator.queue_time_us":16079,"thread_start_us":15950,"threads_started":1}
Jun 20 18:39:00 dist-test-slave-d645 krb5kdc[31545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980740, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM
I20260620 18:39:00.117260 32179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:00.096546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49784 (local address 127.23.115.62:38311)
0620 18:39:00.096976 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:00.096983 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:00.097467 (+ 484us) server_negotiation.cc:407] Connection header received
0620 18:39:00.097747 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:00.097753 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:00.097837 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:00.097932 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:00.100057 (+ 2125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:00.101162 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:00.103194 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:00.103844 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:00.103965 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:00.109539 (+ 5574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:00.109547 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:00.109553 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:00.109600 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:00.113819 (+ 4219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:00.114831 (+ 1012us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:00.114837 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:00.114840 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:00.115045 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:00.115550 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:00.115557 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:00.115561 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:00.116065 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:00.116333 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:00.116645 (+ 312us) server_negotiation.cc:299] Negotiation successful
0620 18:39:00.116899 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":18592,"threads_started":1}
I20260620 18:39:00.121333 31590 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49784:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:39:00.122117 31590 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:39:00.130467 31864 tablet_service.cc:1467] Processing CreateTablet for tablet 1bace2ab1c334930b62d487cfbd79470 (DEFAULT_TABLE table=test-table [id=f5c2f47e6492453781227e8e2089d501]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:39:00.130575 31727 tablet_service.cc:1467] Processing CreateTablet for tablet 1bace2ab1c334930b62d487cfbd79470 (DEFAULT_TABLE table=test-table [id=f5c2f47e6492453781227e8e2089d501]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:39:00.131040 31864 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bace2ab1c334930b62d487cfbd79470. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:39:00.131080 31727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bace2ab1c334930b62d487cfbd79470. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:39:00.138682 32010 tablet_service.cc:1467] Processing CreateTablet for tablet 1bace2ab1c334930b62d487cfbd79470 (DEFAULT_TABLE table=test-table [id=f5c2f47e6492453781227e8e2089d501]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:39:00.139137 32010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bace2ab1c334930b62d487cfbd79470. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:39:00.157404 32188 tablet_bootstrap.cc:492] T 1bace2ab1c334930b62d487cfbd79470 P 526ad504f2c7496a8b22c190efe68692: Bootstrap starting.
I20260620 18:39:00.159580 32188 tablet_bootstrap.cc:654] T 1bace2ab1c334930b62d487cfbd79470 P 526ad504f2c7496a8b22c190efe68692: Neither blocks nor log segments found. Creating new log.
I20260620 18:39:00.160192 32187 tablet_bootstrap.cc:492] T 1bace2ab1c334930b62d487cfbd79470 P 2dac9332ea0345ceb33c40b1779316b2: Bootstrap starting.
I20260620 18:39:00.160830 32186 tablet_bootstrap.cc:492] T 1bace2ab1c334930b62d487cfbd79470 P 011792c5d0a64360a012de15ace8f1c7: Bootstrap starting.
I20260620 18:39:00.164321 32187 tablet_bootstrap.cc:654] T 1bace2ab1c334930b62d487cfbd79470 P 2dac9332ea0345ceb33c40b1779316b2: Neither blocks nor log segments found. Creating new log.
I20260620 18:39:00.166255 32186 tablet_bootstrap.cc:654] T 1bace2ab1c334930b62d487cfbd79470 P 011792c5d0a64360a012de15ace8f1c7: Neither blocks nor log segments found. Creating new log.
I20260620 18:39:00.168893 32188 tablet_bootstrap.cc:492] T 1bace2ab1c334930b62d487cfbd79470 P 526ad504f2c7496a8b22c190efe68692: No bootstrap required, opened a new log
I20260620 18:39:00.169117 32188 ts_tablet_manager.cc:1397] T 1bace2ab1c334930b62d487cfbd79470 P 526ad504f2c7496a8b22c190efe68692: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20260620 18:39:00.170025 32188 raft_consensus.cc:348] T 1bace2ab1c334930b62d487cfbd79470 P 526ad504f2c7496a8b22c190efe68692 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dac9332ea0345ceb33c40b1779316b2" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 35375 } } peers { permanent_uuid: "011792c5d0a64360a012de15ace8f1c7" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 32777 } } peers { permanent_uuid: "526ad504f2c7496a8b22c190efe68692" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 34889 } }
I20260620 18:39:00.171459 32188 raft_consensus.cc:374] T 1bace2ab1c334930b62d487cfbd79470 P 526ad504f2c7496a8b22c190efe68692 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:39:00.171729 32188 raft_consensus.cc:729] T 1bace2ab1c334930b62d487cfbd79470 P 526ad504f2c7496a8b22c190efe68692 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 526ad504f2c7496a8b22c190efe68692, State: Initialized, Role: FOLLOWER
I20260620 18:39:00.172610 32188 consensus_queue.cc:260] T 1bace2ab1c334930b62d487cfbd79470 P 526ad504f2c7496a8b22c190efe68692 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dac9332ea0345ceb33c40b1779316b2" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 35375 } } peers { permanent_uuid: "011792c5d0a64360a012de15ace8f1c7" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 32777 } } peers { permanent_uuid: "526ad504f2c7496a8b22c190efe68692" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 34889 } }
I20260620 18:39:00.176855 32188 ts_tablet_manager.cc:1428] T 1bace2ab1c334930b62d487cfbd79470 P 526ad504f2c7496a8b22c190efe68692: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20260620 18:39:00.183954 32187 tablet_bootstrap.cc:492] T 1bace2ab1c334930b62d487cfbd79470 P 2dac9332ea0345ceb33c40b1779316b2: No bootstrap required, opened a new log
I20260620 18:39:00.184163 32187 ts_tablet_manager.cc:1397] T 1bace2ab1c334930b62d487cfbd79470 P 2dac9332ea0345ceb33c40b1779316b2: Time spent bootstrapping tablet: real 0.024s user 0.000s sys 0.006s
I20260620 18:39:00.185020 32187 raft_consensus.cc:348] T 1bace2ab1c334930b62d487cfbd79470 P 2dac9332ea0345ceb33c40b1779316b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dac9332ea0345ceb33c40b1779316b2" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 35375 } } peers { permanent_uuid: "011792c5d0a64360a012de15ace8f1c7" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 32777 } } peers { permanent_uuid: "526ad504f2c7496a8b22c190efe68692" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 34889 } }
I20260620 18:39:00.185626 32187 raft_consensus.cc:374] T 1bace2ab1c334930b62d487cfbd79470 P 2dac9332ea0345ceb33c40b1779316b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:39:00.185781 32187 raft_consensus.cc:729] T 1bace2ab1c334930b62d487cfbd79470 P 2dac9332ea0345ceb33c40b1779316b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2dac9332ea0345ceb33c40b1779316b2, State: Initialized, Role: FOLLOWER
I20260620 18:39:00.186079 32187 consensus_queue.cc:260] T 1bace2ab1c334930b62d487cfbd79470 P 2dac9332ea0345ceb33c40b1779316b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dac9332ea0345ceb33c40b1779316b2" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 35375 } } peers { permanent_uuid: "011792c5d0a64360a012de15ace8f1c7" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 32777 } } peers { permanent_uuid: "526ad504f2c7496a8b22c190efe68692" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 34889 } }
I20260620 18:39:00.188885 32187 ts_tablet_manager.cc:1428] T 1bace2ab1c334930b62d487cfbd79470 P 2dac9332ea0345ceb33c40b1779316b2: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20260620 18:39:00.192117 32106 raft_consensus.cc:482] T 1bace2ab1c334930b62d487cfbd79470 P 2dac9332ea0345ceb33c40b1779316b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:39:00.192255 32106 raft_consensus.cc:504] T 1bace2ab1c334930b62d487cfbd79470 P 2dac9332ea0345ceb33c40b1779316b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dac9332ea0345ceb33c40b1779316b2" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 35375 } } peers { permanent_uuid: "011792c5d0a64360a012de15ace8f1c7" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 32777 } } peers { permanent_uuid: "526ad504f2c7496a8b22c190efe68692" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 34889 } }
I20260620 18:39:00.192942 32106 leader_election.cc:288] T 1bace2ab1c334930b62d487cfbd79470 P 2dac9332ea0345ceb33c40b1779316b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 011792c5d0a64360a012de15ace8f1c7 (127.23.115.2:32777), 526ad504f2c7496a8b22c190efe68692 (127.23.115.3:34889)
I20260620 18:39:00.193840 32030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bace2ab1c334930b62d487cfbd79470" candidate_uuid: "2dac9332ea0345ceb33c40b1779316b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "526ad504f2c7496a8b22c190efe68692" is_pre_election: true
I20260620 18:39:00.194103 32030 raft_consensus.cc:2454] T 1bace2ab1c334930b62d487cfbd79470 P 526ad504f2c7496a8b22c190efe68692 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2dac9332ea0345ceb33c40b1779316b2 in term 0.
I20260620 18:39:00.194164 31886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bace2ab1c334930b62d487cfbd79470" candidate_uuid: "2dac9332ea0345ceb33c40b1779316b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "011792c5d0a64360a012de15ace8f1c7" is_pre_election: true
I20260620 18:39:00.194499 32186 tablet_bootstrap.cc:492] T 1bace2ab1c334930b62d487cfbd79470 P 011792c5d0a64360a012de15ace8f1c7: No bootstrap required, opened a new log
I20260620 18:39:00.194548 31678 leader_election.cc:302] T 1bace2ab1c334930b62d487cfbd79470 P 2dac9332ea0345ceb33c40b1779316b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2dac9332ea0345ceb33c40b1779316b2, 526ad504f2c7496a8b22c190efe68692; no voters:
I20260620 18:39:00.194741 32186 ts_tablet_manager.cc:1397] T 1bace2ab1c334930b62d487cfbd79470 P 011792c5d0a64360a012de15ace8f1c7: Time spent bootstrapping tablet: real 0.034s user 0.000s sys 0.006s
I20260620 18:39:00.194811 32106 raft_consensus.cc:2781] T 1bace2ab1c334930b62d487cfbd79470 P 2dac9332ea0345ceb33c40b1779316b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:39:00.194955 32106 raft_consensus.cc:482] T 1bace2ab1c334930b62d487cfbd79470 P 2dac9332ea0345ceb33c40b1779316b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:39:00.195067 32106 raft_consensus.cc:3037] T 1bace2ab1c334930b62d487cfbd79470 P 2dac9332ea0345ceb33c40b1779316b2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:39:00.195868 32186 raft_consensus.cc:348] T 1bace2ab1c334930b62d487cfbd79470 P 011792c5d0a64360a012de15ace8f1c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dac9332ea0345ceb33c40b1779316b2" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 35375 } } peers { permanent_uuid: "011792c5d0a64360a012de15ace8f1c7" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 32777 } } peers { permanent_uuid: "526ad504f2c7496a8b22c190efe68692" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 34889 } }
I20260620 18:39:00.196187 32186 raft_consensus.cc:374] T 1bace2ab1c334930b62d487cfbd79470 P 011792c5d0a64360a012de15ace8f1c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:39:00.196336 32186 raft_consensus.cc:729] T 1bace2ab1c334930b62d487cfbd79470 P 011792c5d0a64360a012de15ace8f1c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 011792c5d0a64360a012de15ace8f1c7, State: Initialized, Role: FOLLOWER
I20260620 18:39:00.196684 32186 consensus_queue.cc:260] T 1bace2ab1c334930b62d487cfbd79470 P 011792c5d0a64360a012de15ace8f1c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dac9332ea0345ceb33c40b1779316b2" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 35375 } } peers { permanent_uuid: "011792c5d0a64360a012de15ace8f1c7" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 32777 } } peers { permanent_uuid: "526ad504f2c7496a8b22c190efe68692" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 34889 } }
I20260620 18:39:00.197955 32106 raft_consensus.cc:504] T 1bace2ab1c334930b62d487cfbd79470 P 2dac9332ea0345ceb33c40b1779316b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dac9332ea0345ceb33c40b1779316b2" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 35375 } } peers { permanent_uuid: "011792c5d0a64360a012de15ace8f1c7" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 32777 } } peers { permanent_uuid: "526ad504f2c7496a8b22c190efe68692" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 34889 } }
I20260620 18:39:00.198555 32106 leader_election.cc:288] T 1bace2ab1c334930b62d487cfbd79470 P 2dac9332ea0345ceb33c40b1779316b2 [CANDIDATE]: Term 1 election: Requested vote from peers 011792c5d0a64360a012de15ace8f1c7 (127.23.115.2:32777), 526ad504f2c7496a8b22c190efe68692 (127.23.115.3:34889)
W20260620 18:39:00.199074 31678 leader_election.cc:341] T 1bace2ab1c334930b62d487cfbd79470 P 2dac9332ea0345ceb33c40b1779316b2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 011792c5d0a64360a012de15ace8f1c7 (127.23.115.2:32777): Illegal state: must be running to vote when last-logged opid is not known
I20260620 18:39:00.199726 32030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bace2ab1c334930b62d487cfbd79470" candidate_uuid: "2dac9332ea0345ceb33c40b1779316b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "526ad504f2c7496a8b22c190efe68692"
I20260620 18:39:00.199787 32186 ts_tablet_manager.cc:1428] T 1bace2ab1c334930b62d487cfbd79470 P 011792c5d0a64360a012de15ace8f1c7: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260620 18:39:00.199958 32030 raft_consensus.cc:3037] T 1bace2ab1c334930b62d487cfbd79470 P 526ad504f2c7496a8b22c190efe68692 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:39:00.199965 31886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bace2ab1c334930b62d487cfbd79470" candidate_uuid: "2dac9332ea0345ceb33c40b1779316b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "011792c5d0a64360a012de15ace8f1c7"
I20260620 18:39:00.200145 31886 raft_consensus.cc:3037] T 1bace2ab1c334930b62d487cfbd79470 P 011792c5d0a64360a012de15ace8f1c7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:39:00.202812 32030 raft_consensus.cc:2454] T 1bace2ab1c334930b62d487cfbd79470 P 526ad504f2c7496a8b22c190efe68692 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2dac9332ea0345ceb33c40b1779316b2 in term 1.
I20260620 18:39:00.203118 31886 raft_consensus.cc:2454] T 1bace2ab1c334930b62d487cfbd79470 P 011792c5d0a64360a012de15ace8f1c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2dac9332ea0345ceb33c40b1779316b2 in term 1.
I20260620 18:39:00.203912 31678 leader_election.cc:302] T 1bace2ab1c334930b62d487cfbd79470 P 2dac9332ea0345ceb33c40b1779316b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2dac9332ea0345ceb33c40b1779316b2, 526ad504f2c7496a8b22c190efe68692; no voters:
I20260620 18:39:00.204645 32106 raft_consensus.cc:2781] T 1bace2ab1c334930b62d487cfbd79470 P 2dac9332ea0345ceb33c40b1779316b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:39:00.210177 32106 raft_consensus.cc:686] T 1bace2ab1c334930b62d487cfbd79470 P 2dac9332ea0345ceb33c40b1779316b2 [term 1 LEADER]: Becoming Leader. State: Replica: 2dac9332ea0345ceb33c40b1779316b2, State: Running, Role: LEADER
I20260620 18:39:00.210630 32106 consensus_queue.cc:237] T 1bace2ab1c334930b62d487cfbd79470 P 2dac9332ea0345ceb33c40b1779316b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dac9332ea0345ceb33c40b1779316b2" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 35375 } } peers { permanent_uuid: "011792c5d0a64360a012de15ace8f1c7" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 32777 } } peers { permanent_uuid: "526ad504f2c7496a8b22c190efe68692" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 34889 } }
I20260620 18:39:00.214166 31590 catalog_manager.cc:5495] T 1bace2ab1c334930b62d487cfbd79470 P 2dac9332ea0345ceb33c40b1779316b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2dac9332ea0345ceb33c40b1779316b2 (127.23.115.1). New cstate: current_term: 1 leader_uuid: "2dac9332ea0345ceb33c40b1779316b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dac9332ea0345ceb33c40b1779316b2" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 35375 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "011792c5d0a64360a012de15ace8f1c7" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 32777 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "526ad504f2c7496a8b22c190efe68692" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 34889 } health_report { overall_health: UNKNOWN } } }
I20260620 18:39:00.234400 32195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:00.228965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44156 (local address 127.23.115.1:35375)
0620 18:39:00.229481 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:00.229485 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:00.229503 (+ 18us) server_negotiation.cc:407] Connection header received
0620 18:39:00.229585 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:00.229588 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:00.229693 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:00.229839 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:39:00.230213 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:00.230986 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:00.232235 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:00.232640 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:00.232755 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:00.233173 (+ 418us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:39:00.233753 (+ 580us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:39:00.233898 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:00.233998 (+ 100us) server_negotiation.cc:299] Negotiation successful
0620 18:39:00.234078 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":292,"threads_started":1}
I20260620 18:39:00.238652 32030 raft_consensus.cc:1261] T 1bace2ab1c334930b62d487cfbd79470 P 526ad504f2c7496a8b22c190efe68692 [term 1 FOLLOWER]: Refusing update from remote peer 2dac9332ea0345ceb33c40b1779316b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:39:00.239326 32192 consensus_queue.cc:1035] T 1bace2ab1c334930b62d487cfbd79470 P 2dac9332ea0345ceb33c40b1779316b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "526ad504f2c7496a8b22c190efe68692" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 34889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:39:00.243773 31886 raft_consensus.cc:1261] T 1bace2ab1c334930b62d487cfbd79470 P 011792c5d0a64360a012de15ace8f1c7 [term 1 FOLLOWER]: Refusing update from remote peer 2dac9332ea0345ceb33c40b1779316b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:39:00.244415 32192 consensus_queue.cc:1035] T 1bace2ab1c334930b62d487cfbd79470 P 2dac9332ea0345ceb33c40b1779316b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "011792c5d0a64360a012de15ace8f1c7" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 32777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:39:00.282078 31590 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49784:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:39:00.282333 31590 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49784:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:39:00.285775 31590 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 522d0826a3f64949a4b30e203c4b3a74: Sending DeleteTablet for 3 replicas of tablet 1bace2ab1c334930b62d487cfbd79470
I20260620 18:39:00.286614 31727 tablet_service.cc:1514] Processing DeleteTablet for tablet 1bace2ab1c334930b62d487cfbd79470 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:39:00 UTC) from {username='oryx', principal='oryx/127.23.115.62@KRBTEST.COM'} at 127.0.0.1:44144
I20260620 18:39:00.287091 32169 tablet_replica.cc:330] T 1bace2ab1c334930b62d487cfbd79470 P 2dac9332ea0345ceb33c40b1779316b2: stopping tablet replica
I20260620 18:39:00.287307 32169 raft_consensus.cc:2229] T 1bace2ab1c334930b62d487cfbd79470 P 2dac9332ea0345ceb33c40b1779316b2 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:39:00.287900 32169 raft_consensus.cc:2258] T 1bace2ab1c334930b62d487cfbd79470 P 2dac9332ea0345ceb33c40b1779316b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:39:00.288247 31864 tablet_service.cc:1514] Processing DeleteTablet for tablet 1bace2ab1c334930b62d487cfbd79470 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:39:00 UTC) from {username='oryx', principal='oryx/127.23.115.62@KRBTEST.COM'} at 127.0.0.1:41286
I20260620 18:39:00.288511 32171 tablet_replica.cc:330] T 1bace2ab1c334930b62d487cfbd79470 P 011792c5d0a64360a012de15ace8f1c7: stopping tablet replica
I20260620 18:39:00.288717 32171 raft_consensus.cc:2229] T 1bace2ab1c334930b62d487cfbd79470 P 011792c5d0a64360a012de15ace8f1c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:39:00.288944 32171 raft_consensus.cc:2258] T 1bace2ab1c334930b62d487cfbd79470 P 011792c5d0a64360a012de15ace8f1c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:39:00.289898 32010 tablet_service.cc:1514] Processing DeleteTablet for tablet 1bace2ab1c334930b62d487cfbd79470 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:39:00 UTC) from {username='oryx', principal='oryx/127.23.115.62@KRBTEST.COM'} at 127.0.0.1:55944
I20260620 18:39:00.289974 32169 ts_tablet_manager.cc:1905] T 1bace2ab1c334930b62d487cfbd79470 P 2dac9332ea0345ceb33c40b1779316b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:39:00.290088 32171 ts_tablet_manager.cc:1905] T 1bace2ab1c334930b62d487cfbd79470 P 011792c5d0a64360a012de15ace8f1c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:39:00.290195 32170 tablet_replica.cc:330] T 1bace2ab1c334930b62d487cfbd79470 P 526ad504f2c7496a8b22c190efe68692: stopping tablet replica
I20260620 18:39:00.290378 32170 raft_consensus.cc:2229] T 1bace2ab1c334930b62d487cfbd79470 P 526ad504f2c7496a8b22c190efe68692 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:39:00.290586 32170 raft_consensus.cc:2258] T 1bace2ab1c334930b62d487cfbd79470 P 526ad504f2c7496a8b22c190efe68692 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:39:00.292351 31588 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49784:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:39:00.293157 31588 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:39:00.295150 32170 ts_tablet_manager.cc:1905] T 1bace2ab1c334930b62d487cfbd79470 P 526ad504f2c7496a8b22c190efe68692: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:39:00.296252 32171 ts_tablet_manager.cc:1918] T 1bace2ab1c334930b62d487cfbd79470 P 011792c5d0a64360a012de15ace8f1c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:39:00.296408 32171 log.cc:1196] T 1bace2ab1c334930b62d487cfbd79470 P 011792c5d0a64360a012de15ace8f1c7: Deleting WAL directory at /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980676253068-24012-0/minicluster-data/ts-1/wal/wals/1bace2ab1c334930b62d487cfbd79470
I20260620 18:39:00.296751 32171 ts_tablet_manager.cc:1939] T 1bace2ab1c334930b62d487cfbd79470 P 011792c5d0a64360a012de15ace8f1c7: Deleting consensus metadata
I20260620 18:39:00.298133 32169 ts_tablet_manager.cc:1918] T 1bace2ab1c334930b62d487cfbd79470 P 2dac9332ea0345ceb33c40b1779316b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:39:00.298380 32169 log.cc:1196] T 1bace2ab1c334930b62d487cfbd79470 P 2dac9332ea0345ceb33c40b1779316b2: Deleting WAL directory at /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980676253068-24012-0/minicluster-data/ts-0/wal/wals/1bace2ab1c334930b62d487cfbd79470
I20260620 18:39:00.298846 32169 ts_tablet_manager.cc:1939] T 1bace2ab1c334930b62d487cfbd79470 P 2dac9332ea0345ceb33c40b1779316b2: Deleting consensus metadata
I20260620 18:39:00.299242 31575 catalog_manager.cc:4841] TS 011792c5d0a64360a012de15ace8f1c7 (127.23.115.2:32777): tablet 1bace2ab1c334930b62d487cfbd79470 (table test-table [id=f5c2f47e6492453781227e8e2089d501]) successfully deleted
I20260620 18:39:00.304512 31727 tablet_service.cc:1467] Processing CreateTablet for tablet fde78d3036434b9fb1cbd08a0d06d52e (DEFAULT_TABLE table=test-table [id=9722650587044c1bb29badec0bf77137]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:39:00.304965 31727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fde78d3036434b9fb1cbd08a0d06d52e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:39:00.307613 32170 ts_tablet_manager.cc:1918] T 1bace2ab1c334930b62d487cfbd79470 P 526ad504f2c7496a8b22c190efe68692: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:39:00.307762 32170 log.cc:1196] T 1bace2ab1c334930b62d487cfbd79470 P 526ad504f2c7496a8b22c190efe68692: Deleting WAL directory at /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980676253068-24012-0/minicluster-data/ts-2/wal/wals/1bace2ab1c334930b62d487cfbd79470
I20260620 18:39:00.308204 32170 ts_tablet_manager.cc:1939] T 1bace2ab1c334930b62d487cfbd79470 P 526ad504f2c7496a8b22c190efe68692: Deleting consensus metadata
I20260620 18:39:00.310842 32187 tablet_bootstrap.cc:492] T fde78d3036434b9fb1cbd08a0d06d52e P 2dac9332ea0345ceb33c40b1779316b2: Bootstrap starting.
I20260620 18:39:00.311636 31575 catalog_manager.cc:4841] TS 526ad504f2c7496a8b22c190efe68692 (127.23.115.3:34889): tablet 1bace2ab1c334930b62d487cfbd79470 (table test-table [id=f5c2f47e6492453781227e8e2089d501]) successfully deleted
I20260620 18:39:00.312647 32187 tablet_bootstrap.cc:654] T fde78d3036434b9fb1cbd08a0d06d52e P 2dac9332ea0345ceb33c40b1779316b2: Neither blocks nor log segments found. Creating new log.
I20260620 18:39:00.313393 31576 catalog_manager.cc:4841] TS 2dac9332ea0345ceb33c40b1779316b2 (127.23.115.1:35375): tablet 1bace2ab1c334930b62d487cfbd79470 (table test-table [id=f5c2f47e6492453781227e8e2089d501]) successfully deleted
I20260620 18:39:00.313934 31864 tablet_service.cc:1467] Processing CreateTablet for tablet fde78d3036434b9fb1cbd08a0d06d52e (DEFAULT_TABLE table=test-table [id=9722650587044c1bb29badec0bf77137]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:39:00.314347 31864 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fde78d3036434b9fb1cbd08a0d06d52e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:39:00.318239 32010 tablet_service.cc:1467] Processing CreateTablet for tablet fde78d3036434b9fb1cbd08a0d06d52e (DEFAULT_TABLE table=test-table [id=9722650587044c1bb29badec0bf77137]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:39:00.318663 32010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fde78d3036434b9fb1cbd08a0d06d52e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:39:00.323556 32188 tablet_bootstrap.cc:492] T fde78d3036434b9fb1cbd08a0d06d52e P 526ad504f2c7496a8b22c190efe68692: Bootstrap starting.
I20260620 18:39:00.325206 32188 tablet_bootstrap.cc:654] T fde78d3036434b9fb1cbd08a0d06d52e P 526ad504f2c7496a8b22c190efe68692: Neither blocks nor log segments found. Creating new log.
I20260620 18:39:00.327445 32186 tablet_bootstrap.cc:492] T fde78d3036434b9fb1cbd08a0d06d52e P 011792c5d0a64360a012de15ace8f1c7: Bootstrap starting.
I20260620 18:39:00.328655 32187 tablet_bootstrap.cc:492] T fde78d3036434b9fb1cbd08a0d06d52e P 2dac9332ea0345ceb33c40b1779316b2: No bootstrap required, opened a new log
I20260620 18:39:00.329092 32186 tablet_bootstrap.cc:654] T fde78d3036434b9fb1cbd08a0d06d52e P 011792c5d0a64360a012de15ace8f1c7: Neither blocks nor log segments found. Creating new log.
I20260620 18:39:00.329991 32187 ts_tablet_manager.cc:1397] T fde78d3036434b9fb1cbd08a0d06d52e P 2dac9332ea0345ceb33c40b1779316b2: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.001s
I20260620 18:39:00.330821 32187 raft_consensus.cc:348] T fde78d3036434b9fb1cbd08a0d06d52e P 2dac9332ea0345ceb33c40b1779316b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dac9332ea0345ceb33c40b1779316b2" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 35375 } } peers { permanent_uuid: "011792c5d0a64360a012de15ace8f1c7" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 32777 } } peers { permanent_uuid: "526ad504f2c7496a8b22c190efe68692" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 34889 } }
I20260620 18:39:00.331099 32187 raft_consensus.cc:374] T fde78d3036434b9fb1cbd08a0d06d52e P 2dac9332ea0345ceb33c40b1779316b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:39:00.331182 32187 raft_consensus.cc:729] T fde78d3036434b9fb1cbd08a0d06d52e P 2dac9332ea0345ceb33c40b1779316b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2dac9332ea0345ceb33c40b1779316b2, State: Initialized, Role: FOLLOWER
I20260620 18:39:00.331480 32187 consensus_queue.cc:260] T fde78d3036434b9fb1cbd08a0d06d52e P 2dac9332ea0345ceb33c40b1779316b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dac9332ea0345ceb33c40b1779316b2" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 35375 } } peers { permanent_uuid: "011792c5d0a64360a012de15ace8f1c7" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 32777 } } peers { permanent_uuid: "526ad504f2c7496a8b22c190efe68692" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 34889 } }
I20260620 18:39:00.335486 32187 ts_tablet_manager.cc:1428] T fde78d3036434b9fb1cbd08a0d06d52e P 2dac9332ea0345ceb33c40b1779316b2: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260620 18:39:00.336376 32188 tablet_bootstrap.cc:492] T fde78d3036434b9fb1cbd08a0d06d52e P 526ad504f2c7496a8b22c190efe68692: No bootstrap required, opened a new log
I20260620 18:39:00.336560 32188 ts_tablet_manager.cc:1397] T fde78d3036434b9fb1cbd08a0d06d52e P 526ad504f2c7496a8b22c190efe68692: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.000s
I20260620 18:39:00.337713 32188 raft_consensus.cc:348] T fde78d3036434b9fb1cbd08a0d06d52e P 526ad504f2c7496a8b22c190efe68692 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dac9332ea0345ceb33c40b1779316b2" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 35375 } } peers { permanent_uuid: "011792c5d0a64360a012de15ace8f1c7" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 32777 } } peers { permanent_uuid: "526ad504f2c7496a8b22c190efe68692" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 34889 } }
I20260620 18:39:00.337993 32188 raft_consensus.cc:374] T fde78d3036434b9fb1cbd08a0d06d52e P 526ad504f2c7496a8b22c190efe68692 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:39:00.338078 32188 raft_consensus.cc:729] T fde78d3036434b9fb1cbd08a0d06d52e P 526ad504f2c7496a8b22c190efe68692 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 526ad504f2c7496a8b22c190efe68692, State: Initialized, Role: FOLLOWER
I20260620 18:39:00.338335 32188 consensus_queue.cc:260] T fde78d3036434b9fb1cbd08a0d06d52e P 526ad504f2c7496a8b22c190efe68692 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dac9332ea0345ceb33c40b1779316b2" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 35375 } } peers { permanent_uuid: "011792c5d0a64360a012de15ace8f1c7" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 32777 } } peers { permanent_uuid: "526ad504f2c7496a8b22c190efe68692" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 34889 } }
I20260620 18:39:00.339593 32188 ts_tablet_manager.cc:1428] T fde78d3036434b9fb1cbd08a0d06d52e P 526ad504f2c7496a8b22c190efe68692: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260620 18:39:00.347213 32186 tablet_bootstrap.cc:492] T fde78d3036434b9fb1cbd08a0d06d52e P 011792c5d0a64360a012de15ace8f1c7: No bootstrap required, opened a new log
I20260620 18:39:00.347409 32186 ts_tablet_manager.cc:1397] T fde78d3036434b9fb1cbd08a0d06d52e P 011792c5d0a64360a012de15ace8f1c7: Time spent bootstrapping tablet: real 0.020s user 0.012s sys 0.001s
I20260620 18:39:00.348198 32186 raft_consensus.cc:348] T fde78d3036434b9fb1cbd08a0d06d52e P 011792c5d0a64360a012de15ace8f1c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dac9332ea0345ceb33c40b1779316b2" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 35375 } } peers { permanent_uuid: "011792c5d0a64360a012de15ace8f1c7" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 32777 } } peers { permanent_uuid: "526ad504f2c7496a8b22c190efe68692" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 34889 } }
I20260620 18:39:00.348484 32186 raft_consensus.cc:374] T fde78d3036434b9fb1cbd08a0d06d52e P 011792c5d0a64360a012de15ace8f1c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:39:00.348562 32186 raft_consensus.cc:729] T fde78d3036434b9fb1cbd08a0d06d52e P 011792c5d0a64360a012de15ace8f1c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 011792c5d0a64360a012de15ace8f1c7, State: Initialized, Role: FOLLOWER
I20260620 18:39:00.348827 32186 consensus_queue.cc:260] T fde78d3036434b9fb1cbd08a0d06d52e P 011792c5d0a64360a012de15ace8f1c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dac9332ea0345ceb33c40b1779316b2" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 35375 } } peers { permanent_uuid: "011792c5d0a64360a012de15ace8f1c7" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 32777 } } peers { permanent_uuid: "526ad504f2c7496a8b22c190efe68692" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 34889 } }
I20260620 18:39:00.350103 32186 ts_tablet_manager.cc:1428] T fde78d3036434b9fb1cbd08a0d06d52e P 011792c5d0a64360a012de15ace8f1c7: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260620 18:39:00.554358 32198 raft_consensus.cc:482] T fde78d3036434b9fb1cbd08a0d06d52e P 2dac9332ea0345ceb33c40b1779316b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:39:00.554622 32198 raft_consensus.cc:504] T fde78d3036434b9fb1cbd08a0d06d52e P 2dac9332ea0345ceb33c40b1779316b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dac9332ea0345ceb33c40b1779316b2" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 35375 } } peers { permanent_uuid: "011792c5d0a64360a012de15ace8f1c7" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 32777 } } peers { permanent_uuid: "526ad504f2c7496a8b22c190efe68692" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 34889 } }
I20260620 18:39:00.555445 32198 leader_election.cc:288] T fde78d3036434b9fb1cbd08a0d06d52e P 2dac9332ea0345ceb33c40b1779316b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 011792c5d0a64360a012de15ace8f1c7 (127.23.115.2:32777), 526ad504f2c7496a8b22c190efe68692 (127.23.115.3:34889)
I20260620 18:39:00.556180 31886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fde78d3036434b9fb1cbd08a0d06d52e" candidate_uuid: "2dac9332ea0345ceb33c40b1779316b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "011792c5d0a64360a012de15ace8f1c7" is_pre_election: true
I20260620 18:39:00.556233 32030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fde78d3036434b9fb1cbd08a0d06d52e" candidate_uuid: "2dac9332ea0345ceb33c40b1779316b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "526ad504f2c7496a8b22c190efe68692" is_pre_election: true
I20260620 18:39:00.556494 32030 raft_consensus.cc:2454] T fde78d3036434b9fb1cbd08a0d06d52e P 526ad504f2c7496a8b22c190efe68692 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2dac9332ea0345ceb33c40b1779316b2 in term 0.
I20260620 18:39:00.556485 31886 raft_consensus.cc:2454] T fde78d3036434b9fb1cbd08a0d06d52e P 011792c5d0a64360a012de15ace8f1c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2dac9332ea0345ceb33c40b1779316b2 in term 0.
I20260620 18:39:00.557173 31678 leader_election.cc:302] T fde78d3036434b9fb1cbd08a0d06d52e P 2dac9332ea0345ceb33c40b1779316b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 011792c5d0a64360a012de15ace8f1c7, 2dac9332ea0345ceb33c40b1779316b2; no voters:
I20260620 18:39:00.557583 32198 raft_consensus.cc:2781] T fde78d3036434b9fb1cbd08a0d06d52e P 2dac9332ea0345ceb33c40b1779316b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:39:00.557866 32198 raft_consensus.cc:482] T fde78d3036434b9fb1cbd08a0d06d52e P 2dac9332ea0345ceb33c40b1779316b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:39:00.558178 32198 raft_consensus.cc:3037] T fde78d3036434b9fb1cbd08a0d06d52e P 2dac9332ea0345ceb33c40b1779316b2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:39:00.560722 32198 raft_consensus.cc:504] T fde78d3036434b9fb1cbd08a0d06d52e P 2dac9332ea0345ceb33c40b1779316b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dac9332ea0345ceb33c40b1779316b2" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 35375 } } peers { permanent_uuid: "011792c5d0a64360a012de15ace8f1c7" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 32777 } } peers { permanent_uuid: "526ad504f2c7496a8b22c190efe68692" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 34889 } }
I20260620 18:39:00.561307 32198 leader_election.cc:288] T fde78d3036434b9fb1cbd08a0d06d52e P 2dac9332ea0345ceb33c40b1779316b2 [CANDIDATE]: Term 1 election: Requested vote from peers 011792c5d0a64360a012de15ace8f1c7 (127.23.115.2:32777), 526ad504f2c7496a8b22c190efe68692 (127.23.115.3:34889)
I20260620 18:39:00.561844 32030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fde78d3036434b9fb1cbd08a0d06d52e" candidate_uuid: "2dac9332ea0345ceb33c40b1779316b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "526ad504f2c7496a8b22c190efe68692"
I20260620 18:39:00.561812 31886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fde78d3036434b9fb1cbd08a0d06d52e" candidate_uuid: "2dac9332ea0345ceb33c40b1779316b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "011792c5d0a64360a012de15ace8f1c7"
I20260620 18:39:00.562029 31886 raft_consensus.cc:3037] T fde78d3036434b9fb1cbd08a0d06d52e P 011792c5d0a64360a012de15ace8f1c7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:39:00.562049 32030 raft_consensus.cc:3037] T fde78d3036434b9fb1cbd08a0d06d52e P 526ad504f2c7496a8b22c190efe68692 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:39:00.564208 32030 raft_consensus.cc:2454] T fde78d3036434b9fb1cbd08a0d06d52e P 526ad504f2c7496a8b22c190efe68692 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2dac9332ea0345ceb33c40b1779316b2 in term 1.
I20260620 18:39:00.564592 31678 leader_election.cc:302] T fde78d3036434b9fb1cbd08a0d06d52e P 2dac9332ea0345ceb33c40b1779316b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2dac9332ea0345ceb33c40b1779316b2, 526ad504f2c7496a8b22c190efe68692; no voters:
I20260620 18:39:00.564855 32198 raft_consensus.cc:2781] T fde78d3036434b9fb1cbd08a0d06d52e P 2dac9332ea0345ceb33c40b1779316b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:39:00.564926 31886 raft_consensus.cc:2454] T fde78d3036434b9fb1cbd08a0d06d52e P 011792c5d0a64360a012de15ace8f1c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2dac9332ea0345ceb33c40b1779316b2 in term 1.
I20260620 18:39:00.565112 32198 raft_consensus.cc:686] T fde78d3036434b9fb1cbd08a0d06d52e P 2dac9332ea0345ceb33c40b1779316b2 [term 1 LEADER]: Becoming Leader. State: Replica: 2dac9332ea0345ceb33c40b1779316b2, State: Running, Role: LEADER
I20260620 18:39:00.565446 32198 consensus_queue.cc:237] T fde78d3036434b9fb1cbd08a0d06d52e P 2dac9332ea0345ceb33c40b1779316b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dac9332ea0345ceb33c40b1779316b2" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 35375 } } peers { permanent_uuid: "011792c5d0a64360a012de15ace8f1c7" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 32777 } } peers { permanent_uuid: "526ad504f2c7496a8b22c190efe68692" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 34889 } }
I20260620 18:39:00.568073 31590 catalog_manager.cc:5495] T fde78d3036434b9fb1cbd08a0d06d52e P 2dac9332ea0345ceb33c40b1779316b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2dac9332ea0345ceb33c40b1779316b2 (127.23.115.1). New cstate: current_term: 1 leader_uuid: "2dac9332ea0345ceb33c40b1779316b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dac9332ea0345ceb33c40b1779316b2" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 35375 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "011792c5d0a64360a012de15ace8f1c7" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 32777 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "526ad504f2c7496a8b22c190efe68692" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 34889 } health_report { overall_health: UNKNOWN } } }
I20260620 18:39:00.652995 32207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:00.646304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.2:32777 (local address 127.23.115.1:45905)
0620 18:39:00.647036 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:00.647057 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:00.647214 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:00.647647 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:00.647650 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:00.647669 (+ 19us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 18:39:00.648028 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:00.648037 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:00.649401 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:00.649405 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:00.650680 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:00.650693 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:00.651767 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:00.651774 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:00.651946 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:00.652006 (+ 60us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 18:39:00.652473 (+ 467us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 18:39:00.652484 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 18:39:00.652561 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 18:39:00.652678 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":276,"threads_started":1}
I20260620 18:39:00.653014 32208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:00.646409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.1:45905 (local address 127.23.115.2:32777)
0620 18:39:00.647107 (+ 698us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:00.647113 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:00.647135 (+ 22us) server_negotiation.cc:407] Connection header received
0620 18:39:00.647413 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:00.647418 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:00.647506 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:00.647612 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:39:00.648205 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:00.649253 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:00.650828 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:00.651553 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:00.651697 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:00.652186 (+ 489us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:39:00.652318 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:39:00.652423 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:00.652597 (+ 174us) server_negotiation.cc:299] Negotiation successful
0620 18:39:00.652681 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":397,"threads_started":1}
I20260620 18:39:00.662348 32030 raft_consensus.cc:1261] T fde78d3036434b9fb1cbd08a0d06d52e P 526ad504f2c7496a8b22c190efe68692 [term 1 FOLLOWER]: Refusing update from remote peer 2dac9332ea0345ceb33c40b1779316b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:39:00.662397 31886 raft_consensus.cc:1261] T fde78d3036434b9fb1cbd08a0d06d52e P 011792c5d0a64360a012de15ace8f1c7 [term 1 FOLLOWER]: Refusing update from remote peer 2dac9332ea0345ceb33c40b1779316b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:39:00.663081 32198 consensus_queue.cc:1035] T fde78d3036434b9fb1cbd08a0d06d52e P 2dac9332ea0345ceb33c40b1779316b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "011792c5d0a64360a012de15ace8f1c7" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 32777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:39:00.663398 32192 consensus_queue.cc:1035] T fde78d3036434b9fb1cbd08a0d06d52e P 2dac9332ea0345ceb33c40b1779316b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "526ad504f2c7496a8b22c190efe68692" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 34889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:39:00.693791 32195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:00.687279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.2:46519 (local address 127.23.115.1:35375)
0620 18:39:00.687469 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:00.687476 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:00.687904 (+ 428us) server_negotiation.cc:407] Connection header received
0620 18:39:00.688042 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:00.688047 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:00.688135 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:00.688232 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:39:00.688608 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:00.689721 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:00.691578 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:00.692288 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:00.692411 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:00.692985 (+ 574us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:39:00.693105 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:39:00.693191 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:00.693394 (+ 203us) server_negotiation.cc:299] Negotiation successful
0620 18:39:00.693483 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 18:39:00.694372 32213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:00.687218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.1:35375 (local address 127.23.115.2:46519)
0620 18:39:00.687758 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:00.687777 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:00.687931 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:00.688249 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:00.688253 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:00.688265 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 18:39:00.688485 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:00.688495 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:00.689843 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:00.689849 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:00.691415 (+ 1566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:00.691431 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:00.692569 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:00.692576 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:00.692743 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:00.692800 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 18:39:00.693228 (+ 428us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 18:39:00.693240 (+ 12us) client_negotiation.cc:769] Sending connection context
0620 18:39:00.693997 (+ 757us) client_negotiation.cc:240] Negotiation successful
0620 18:39:00.694122 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":268,"threads_started":1}
I20260620 18:39:00.726004 31589 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49784:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:39:00.726257 31589 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49784:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:39:00.729328 31589 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 522d0826a3f64949a4b30e203c4b3a74: Sending DeleteTablet for 3 replicas of tablet fde78d3036434b9fb1cbd08a0d06d52e
I20260620 18:39:00.730019 31727 tablet_service.cc:1514] Processing DeleteTablet for tablet fde78d3036434b9fb1cbd08a0d06d52e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:39:00 UTC) from {username='oryx', principal='oryx/127.23.115.62@KRBTEST.COM'} at 127.0.0.1:44144
I20260620 18:39:00.730232 32010 tablet_service.cc:1514] Processing DeleteTablet for tablet fde78d3036434b9fb1cbd08a0d06d52e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:39:00 UTC) from {username='oryx', principal='oryx/127.23.115.62@KRBTEST.COM'} at 127.0.0.1:55944
I20260620 18:39:00.730242 31864 tablet_service.cc:1514] Processing DeleteTablet for tablet fde78d3036434b9fb1cbd08a0d06d52e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:39:00 UTC) from {username='oryx', principal='oryx/127.23.115.62@KRBTEST.COM'} at 127.0.0.1:41286
I20260620 18:39:00.730381 32169 tablet_replica.cc:330] T fde78d3036434b9fb1cbd08a0d06d52e P 2dac9332ea0345ceb33c40b1779316b2: stopping tablet replica
I20260620 18:39:00.730497 32171 tablet_replica.cc:330] T fde78d3036434b9fb1cbd08a0d06d52e P 011792c5d0a64360a012de15ace8f1c7: stopping tablet replica
I20260620 18:39:00.730540 32170 tablet_replica.cc:330] T fde78d3036434b9fb1cbd08a0d06d52e P 526ad504f2c7496a8b22c190efe68692: stopping tablet replica
I20260620 18:39:00.730549 32169 raft_consensus.cc:2229] T fde78d3036434b9fb1cbd08a0d06d52e P 2dac9332ea0345ceb33c40b1779316b2 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:39:00.730643 32171 raft_consensus.cc:2229] T fde78d3036434b9fb1cbd08a0d06d52e P 011792c5d0a64360a012de15ace8f1c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:39:00.730716 32170 raft_consensus.cc:2229] T fde78d3036434b9fb1cbd08a0d06d52e P 526ad504f2c7496a8b22c190efe68692 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:39:00.730849 32171 raft_consensus.cc:2258] T fde78d3036434b9fb1cbd08a0d06d52e P 011792c5d0a64360a012de15ace8f1c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:39:00.730863 32169 raft_consensus.cc:2258] T fde78d3036434b9fb1cbd08a0d06d52e P 2dac9332ea0345ceb33c40b1779316b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:39:00.730892 32170 raft_consensus.cc:2258] T fde78d3036434b9fb1cbd08a0d06d52e P 526ad504f2c7496a8b22c190efe68692 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:39:00.731976 32170 ts_tablet_manager.cc:1905] T fde78d3036434b9fb1cbd08a0d06d52e P 526ad504f2c7496a8b22c190efe68692: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:39:00.731978 32169 ts_tablet_manager.cc:1905] T fde78d3036434b9fb1cbd08a0d06d52e P 2dac9332ea0345ceb33c40b1779316b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:39:00.732637 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 31661
I20260620 18:39:00.738575 32170 ts_tablet_manager.cc:1918] T fde78d3036434b9fb1cbd08a0d06d52e P 526ad504f2c7496a8b22c190efe68692: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:39:00.738633 32169 ts_tablet_manager.cc:1918] T fde78d3036434b9fb1cbd08a0d06d52e P 2dac9332ea0345ceb33c40b1779316b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:39:00.738721 32170 log.cc:1196] T fde78d3036434b9fb1cbd08a0d06d52e P 526ad504f2c7496a8b22c190efe68692: Deleting WAL directory at /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980676253068-24012-0/minicluster-data/ts-2/wal/wals/fde78d3036434b9fb1cbd08a0d06d52e
I20260620 18:39:00.738747 32169 log.cc:1196] T fde78d3036434b9fb1cbd08a0d06d52e P 2dac9332ea0345ceb33c40b1779316b2: Deleting WAL directory at /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980676253068-24012-0/minicluster-data/ts-0/wal/wals/fde78d3036434b9fb1cbd08a0d06d52e
I20260620 18:39:00.739181 32170 ts_tablet_manager.cc:1939] T fde78d3036434b9fb1cbd08a0d06d52e P 526ad504f2c7496a8b22c190efe68692: Deleting consensus metadata
I20260620 18:39:00.739204 32169 ts_tablet_manager.cc:1939] T fde78d3036434b9fb1cbd08a0d06d52e P 2dac9332ea0345ceb33c40b1779316b2: Deleting consensus metadata
I20260620 18:39:00.740124 32171 ts_tablet_manager.cc:1905] T fde78d3036434b9fb1cbd08a0d06d52e P 011792c5d0a64360a012de15ace8f1c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:39:00.740195 31576 catalog_manager.cc:4841] TS 2dac9332ea0345ceb33c40b1779316b2 (127.23.115.1:35375): tablet fde78d3036434b9fb1cbd08a0d06d52e (table test-table [id=9722650587044c1bb29badec0bf77137]) successfully deleted
I20260620 18:39:00.743098 31575 catalog_manager.cc:4841] TS 526ad504f2c7496a8b22c190efe68692 (127.23.115.3:34889): tablet fde78d3036434b9fb1cbd08a0d06d52e (table test-table [id=9722650587044c1bb29badec0bf77137]) successfully deleted
I20260620 18:39:00.744541 32171 ts_tablet_manager.cc:1918] T fde78d3036434b9fb1cbd08a0d06d52e P 011792c5d0a64360a012de15ace8f1c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:39:00.744650 32171 log.cc:1196] T fde78d3036434b9fb1cbd08a0d06d52e P 011792c5d0a64360a012de15ace8f1c7: Deleting WAL directory at /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980676253068-24012-0/minicluster-data/ts-1/wal/wals/fde78d3036434b9fb1cbd08a0d06d52e
I20260620 18:39:00.744971 32171 ts_tablet_manager.cc:1939] T fde78d3036434b9fb1cbd08a0d06d52e P 011792c5d0a64360a012de15ace8f1c7: Deleting consensus metadata
I20260620 18:39:00.745863 31575 catalog_manager.cc:4841] TS 011792c5d0a64360a012de15ace8f1c7 (127.23.115.2:32777): tablet fde78d3036434b9fb1cbd08a0d06d52e (table test-table [id=9722650587044c1bb29badec0bf77137]) successfully deleted
Jun 20 18:39:00 dist-test-slave-d645 krb5kdc[31545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980740, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.115.1@KRBTEST.COM
I20260620 18:39:00.757915 32195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:00.734317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44166 (local address 127.23.115.1:35375)
0620 18:39:00.735536 (+ 1219us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:00.735542 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:00.743500 (+ 7958us) server_negotiation.cc:407] Connection header received
0620 18:39:00.743559 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:00.743563 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:00.743642 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:00.743740 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:00.744941 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:00.745657 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:00.746897 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:00.747307 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:00.747407 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:00.751209 (+ 3802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:00.751224 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:00.751244 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:00.751286 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:00.754870 (+ 3584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:00.755603 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:00.755614 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:00.755618 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:00.755765 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:00.756251 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:00.756256 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:00.756260 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:00.756686 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:00.756986 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:00.757331 (+ 345us) server_negotiation.cc:299] Negotiation successful
0620 18:39:00.757551 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1101}
I20260620 18:39:00.758548 31792 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:39:00.897305 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 31661
I20260620 18:39:00.917083 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 31801
Jun 20 18:39:00 dist-test-slave-d645 krb5kdc[31545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980740, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.115.2@KRBTEST.COM
I20260620 18:39:00.932590 32208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:00.917922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41320 (local address 127.23.115.2:32777)
0620 18:39:00.918068 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:00.918073 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:00.918089 (+ 16us) server_negotiation.cc:407] Connection header received
0620 18:39:00.918147 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:00.918151 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:00.918210 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:00.918325 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:00.919686 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:00.920396 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:00.921857 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:00.922279 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:00.922393 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:00.926790 (+ 4397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:00.926807 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:00.926825 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:00.926872 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:00.929918 (+ 3046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:00.930583 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:00.930590 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:00.930593 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:00.930728 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:00.931117 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:00.931121 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:00.931123 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:00.931473 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:00.931761 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:00.932076 (+ 315us) server_negotiation.cc:299] Negotiation successful
0620 18:39:00.932298 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 18:39:00.933283 31931 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:39:01.077580 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 31801
I20260620 18:39:01.098168 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 31940
Jun 20 18:39:01 dist-test-slave-d645 krb5kdc[31545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980740, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.115.3@KRBTEST.COM
I20260620 18:39:01.113026 32218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:01.099012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55964 (local address 127.23.115.3:34889)
0620 18:39:01.099469 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:01.099474 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:01.099490 (+ 16us) server_negotiation.cc:407] Connection header received
0620 18:39:01.099543 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:01.099546 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:01.099613 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:01.099708 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:01.100967 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:01.101725 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:01.103127 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:01.103647 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:01.103742 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:01.107935 (+ 4193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:01.107949 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:01.107967 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:01.108001 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:01.110522 (+ 2521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:01.111249 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:01.111257 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:01.111259 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:01.111405 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:01.111771 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:01.111773 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:01.111775 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:01.112098 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:01.112320 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:01.112608 (+ 288us) server_negotiation.cc:299] Negotiation successful
0620 18:39:01.112768 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":244,"threads_started":1}
I20260620 18:39:01.113649 32070 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:39:01.235548 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 31940
I20260620 18:39:01.252954 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 31558
I20260620 18:39:01.254171 31630 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:39:01.373629 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 31558
2026-06-20T18:39:01Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (6856 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980676253068-24012-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:39:01 dist-test-slave-d645 krb5kdc[32223](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:39:01 dist-test-slave-d645 krb5kdc[32223](info): set up 2 sockets
Jun 20 18:39:01 dist-test-slave-d645 krb5kdc[32223](info): commencing operation
krb5kdc: starting...
W20260620 18:39:02.395969 24012 mini_kdc.cc:121] Time spent starting KDC: real 0.982s user 0.000s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980676253068-24012-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980676253068-24012-0/krb5kdc/test-user.keytab.
Jun 20 18:39:02 dist-test-slave-d645 krb5kdc[32223](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:39:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:39:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.115.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.115.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.115.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab.
Entry for principal kudu/127.23.115.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab.
I20260620 18:39:02.608063 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980676253068-24012-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980676253068-24012-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980676253068-24012-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.115.62:33941
--webserver_interface=127.23.115.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab
--principal=kudu/127.23.115.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:35021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.115.62:33941
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:39:02.927512 32236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:39:02.927822 32236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:39:02.927954 32236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:39:02.935216 32236 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:39:02.935335 32236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:39:02.935416 32236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:39:02.935489 32236 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:39:02.935550 32236 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:39:02.945477 32236 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:35021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980676253068-24012-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980676253068-24012-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.115.62:33941
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.115.62:33941
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.115.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980676253068-24012-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:39:02.946776 32236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:39:02.948061 32236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:39:02.959542 32244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:02.960043 32241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:02.960147 32242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:02.960448 32236 server_base.cc:1034] running on GCE node
I20260620 18:39:02.961234 32236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:39:02.962688 32236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:39:02.963896 32236 hybrid_clock.cc:648] HybridClock initialized: now 1781980742963825 us; error 55 us; skew 500 ppm
I20260620 18:39:02.964231 32236 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-20T18:39:03Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1664 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:39:03 dist-test-slave-d645 krb5kdc[32250](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:39:03 dist-test-slave-d645 krb5kdc[32250](info): set up 2 sockets
Jun 20 18:39:03 dist-test-slave-d645 krb5kdc[32250](info): commencing operation
krb5kdc: starting...
W20260620 18:39:04.059818 24012 mini_kdc.cc:121] Time spent starting KDC: real 0.981s user 0.000s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/krb5kdc/test-user.keytab.
Jun 20 18:39:04 dist-test-slave-d645 krb5kdc[32250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980744, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:39:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:39:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.115.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.115.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.115.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab.
Entry for principal kudu/127.23.115.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab.
I20260620 18:39:04.271356 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.115.62:43103
--webserver_interface=127.23.115.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab
--principal=kudu/127.23.115.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:38503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.115.62:43103
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:39:04.603879 32263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:39:04.604183 32263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:39:04.604321 32263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:39:04.611577 32263 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:39:04.611694 32263 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:39:04.611773 32263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:39:04.611850 32263 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:39:04.611913 32263 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:39:04.621701 32263 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:38503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.115.62:43103
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab
--principal=kudu/127.23.115.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.115.62:43103
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.115.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:39:04.623018 32263 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:39:04.624277 32263 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:39:04.635776 32269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:04.635824 32268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:04.636160 32271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:04.636423 32263 server_base.cc:1034] running on GCE node
I20260620 18:39:04.637151 32263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:39:04.638556 32263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:39:04.639752 32263 hybrid_clock.cc:648] HybridClock initialized: now 1781980744639695 us; error 52 us; skew 500 ppm
I20260620 18:39:04.640050 32263 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:39:04 dist-test-slave-d645 krb5kdc[32250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980744, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:39:04.642899 32263 init.cc:377] Logged in from keytab as kudu/127.23.115.62@KRBTEST.COM (short username kudu)
I20260620 18:39:04.644361 32263 webserver.cc:460] Webserver started at http://127.23.115.62:37549/ using document root <none> and password file <none>
I20260620 18:39:04.644994 32263 fs_manager.cc:359] Metadata directory not provided
I20260620 18:39:04.645087 32263 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:39:04.645339 32263 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:39:04.648141 32263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/master-0/data/instance:
uuid: "3cd2627faa2e4667b6c5f8d7e6408827"
format_stamp: "Formatted at 2026-06-20 18:39:04 on dist-test-slave-d645"
server_key: "b5a492cd09578221b07adeed77ae87eb"
server_key_iv: "a59ad8582878518b1d1d65d8b4b4cb5b"
server_key_version: "encryptionkey@0"
I20260620 18:39:04.648823 32263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/master-0/wal/instance:
uuid: "3cd2627faa2e4667b6c5f8d7e6408827"
format_stamp: "Formatted at 2026-06-20 18:39:04 on dist-test-slave-d645"
server_key: "b5a492cd09578221b07adeed77ae87eb"
server_key_iv: "a59ad8582878518b1d1d65d8b4b4cb5b"
server_key_version: "encryptionkey@0"
I20260620 18:39:04.653738 32263 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 18:39:04.657565 32278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:04.659065 32263 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:39:04.659219 32263 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/master-0/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/master-0/wal
uuid: "3cd2627faa2e4667b6c5f8d7e6408827"
format_stamp: "Formatted at 2026-06-20 18:39:04 on dist-test-slave-d645"
server_key: "b5a492cd09578221b07adeed77ae87eb"
server_key_iv: "a59ad8582878518b1d1d65d8b4b4cb5b"
server_key_version: "encryptionkey@0"
I20260620 18:39:04.659420 32263 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:39:04.685585 32263 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:39:04.689509 32263 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:39:04.689738 32263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:39:04.708423 32263 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.62:43103
I20260620 18:39:04.708425 32329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.62:43103 every 8 connection(s)
I20260620 18:39:04.710198 32263 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
I20260620 18:39:04.713302 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 32263
I20260620 18:39:04.713531 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/master-0/wal/instance
I20260620 18:39:04.714327 24012 external_mini_cluster.cc:1428] Setting key 9f8eb8e7237da80b9a50f4c75d84adc1
I20260620 18:39:04.714445 32330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:39:04.728639 32330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3cd2627faa2e4667b6c5f8d7e6408827: Bootstrap starting.
Jun 20 18:39:04 dist-test-slave-d645 krb5kdc[32250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980744, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.115.62@KRBTEST.COM
I20260620 18:39:04.732271 32330 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3cd2627faa2e4667b6c5f8d7e6408827: Neither blocks nor log segments found. Creating new log.
I20260620 18:39:04.733332 32330 log.cc:824] T 00000000000000000000000000000000 P 3cd2627faa2e4667b6c5f8d7e6408827: Log is configured to *not* fsync() on all Append() calls
I20260620 18:39:04.736569 32330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3cd2627faa2e4667b6c5f8d7e6408827: No bootstrap required, opened a new log
I20260620 18:39:04.737802 32333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:04.716261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52938 (local address 127.23.115.62:43103)
0620 18:39:04.716999 (+ 738us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:04.717010 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:04.717049 (+ 39us) server_negotiation.cc:407] Connection header received
0620 18:39:04.719146 (+ 2097us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:04.719169 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:04.719561 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:04.719882 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:04.721581 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:04.722898 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:04.724664 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:04.725439 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:04.725589 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:04.730393 (+ 4804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:04.730402 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:04.730417 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:04.730457 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:04.733535 (+ 3078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:04.734403 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:04.734413 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:04.734425 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:04.734641 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:04.735113 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:04.735117 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:04.735119 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:04.735671 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:04.735889 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:04.736416 (+ 527us) server_negotiation.cc:299] Negotiation successful
0620 18:39:04.736775 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":316,"threads_started":1}
I20260620 18:39:04.744009 32330 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3cd2627faa2e4667b6c5f8d7e6408827 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cd2627faa2e4667b6c5f8d7e6408827" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 43103 } }
I20260620 18:39:04.744356 32330 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3cd2627faa2e4667b6c5f8d7e6408827 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:39:04.744462 32330 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3cd2627faa2e4667b6c5f8d7e6408827 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cd2627faa2e4667b6c5f8d7e6408827, State: Initialized, Role: FOLLOWER
I20260620 18:39:04.744978 32330 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3cd2627faa2e4667b6c5f8d7e6408827 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cd2627faa2e4667b6c5f8d7e6408827" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 43103 } }
I20260620 18:39:04.745190 32330 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3cd2627faa2e4667b6c5f8d7e6408827 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:39:04.745313 32330 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3cd2627faa2e4667b6c5f8d7e6408827 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:39:04.745468 32330 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3cd2627faa2e4667b6c5f8d7e6408827 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:39:04.747627 32330 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3cd2627faa2e4667b6c5f8d7e6408827 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cd2627faa2e4667b6c5f8d7e6408827" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 43103 } }
I20260620 18:39:04.748147 32330 leader_election.cc:302] T 00000000000000000000000000000000 P 3cd2627faa2e4667b6c5f8d7e6408827 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3cd2627faa2e4667b6c5f8d7e6408827; no voters:
I20260620 18:39:04.748705 32330 leader_election.cc:288] T 00000000000000000000000000000000 P 3cd2627faa2e4667b6c5f8d7e6408827 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:39:04.748886 32335 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3cd2627faa2e4667b6c5f8d7e6408827 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:39:04.749848 32335 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3cd2627faa2e4667b6c5f8d7e6408827 [term 1 LEADER]: Becoming Leader. State: Replica: 3cd2627faa2e4667b6c5f8d7e6408827, State: Running, Role: LEADER
I20260620 18:39:04.750379 32335 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3cd2627faa2e4667b6c5f8d7e6408827 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cd2627faa2e4667b6c5f8d7e6408827" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 43103 } }
I20260620 18:39:04.750840 32330 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3cd2627faa2e4667b6c5f8d7e6408827 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:39:04.753217 32336 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cd2627faa2e4667b6c5f8d7e6408827 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3cd2627faa2e4667b6c5f8d7e6408827" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cd2627faa2e4667b6c5f8d7e6408827" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 43103 } } }
I20260620 18:39:04.753249 32337 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cd2627faa2e4667b6c5f8d7e6408827 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3cd2627faa2e4667b6c5f8d7e6408827. Latest consensus state: current_term: 1 leader_uuid: "3cd2627faa2e4667b6c5f8d7e6408827" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cd2627faa2e4667b6c5f8d7e6408827" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 43103 } } }
I20260620 18:39:04.753608 32336 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cd2627faa2e4667b6c5f8d7e6408827 [sys.catalog]: This master's current role is: LEADER
I20260620 18:39:04.753609 32337 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cd2627faa2e4667b6c5f8d7e6408827 [sys.catalog]: This master's current role is: LEADER
I20260620 18:39:04.755148 32341 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:39:04.760376 32341 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:39:04.769163 32341 catalog_manager.cc:1347] Generated new cluster ID: 29233afe22974cbf97c8c083df9c2306
I20260620 18:39:04.769284 32341 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:39:04.793363 32341 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:39:04.794652 32341 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:39:04.805115 32341 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3cd2627faa2e4667b6c5f8d7e6408827: Generated new TSK 0
I20260620 18:39:04.805915 32341 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.115.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.115.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.115.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.1.keytab.
Entry for principal kudu/127.23.115.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.1.keytab.
I20260620 18:39:04.866703 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.115.1:0
--local_ip_for_outbound_sockets=127.23.115.1
--webserver_interface=127.23.115.1
--webserver_port=0
--tserver_master_addrs=127.23.115.62:43103
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.1.keytab
--principal=kudu/127.23.115.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:38503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:39:05.185664 32356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:39:05.185946 32356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:39:05.186069 32356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:39:05.193240 32356 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:39:05.193372 32356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:39:05.193523 32356 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.115.1
I20260620 18:39:05.203218 32356 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:38503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.1.keytab
--principal=kudu/127.23.115.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.115.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.115.1
--webserver_port=0
--tserver_master_addrs=127.23.115.62:43103
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.115.1
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:39:05.204478 32356 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:39:05.205761 32356 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:39:05.218228 32362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:05.218353 32361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:05.218576 32364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:05.219158 32356 server_base.cc:1034] running on GCE node
I20260620 18:39:05.219704 32356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:39:05.220633 32356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:39:05.221856 32356 hybrid_clock.cc:648] HybridClock initialized: now 1781980745221804 us; error 40 us; skew 500 ppm
I20260620 18:39:05.222190 32356 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:39:05 dist-test-slave-d645 krb5kdc[32250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980745, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:39:05.225394 32356 init.cc:377] Logged in from keytab as kudu/127.23.115.1@KRBTEST.COM (short username kudu)
I20260620 18:39:05.226854 32356 webserver.cc:460] Webserver started at http://127.23.115.1:34187/ using document root <none> and password file <none>
I20260620 18:39:05.227545 32356 fs_manager.cc:359] Metadata directory not provided
I20260620 18:39:05.227659 32356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:39:05.227964 32356 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:39:05.231159 32356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/ts-0/data/instance:
uuid: "21c659b9cfe24c87a2fe9706a2c0e254"
format_stamp: "Formatted at 2026-06-20 18:39:05 on dist-test-slave-d645"
server_key: "e3a6049f03d87e31b646183014021f6b"
server_key_iv: "c67f120301731522179b622269243275"
server_key_version: "encryptionkey@0"
I20260620 18:39:05.232012 32356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/ts-0/wal/instance:
uuid: "21c659b9cfe24c87a2fe9706a2c0e254"
format_stamp: "Formatted at 2026-06-20 18:39:05 on dist-test-slave-d645"
server_key: "e3a6049f03d87e31b646183014021f6b"
server_key_iv: "c67f120301731522179b622269243275"
server_key_version: "encryptionkey@0"
I20260620 18:39:05.237095 32356 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260620 18:39:05.240638 32371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:05.242136 32356 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 18:39:05.242314 32356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/ts-0/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/ts-0/wal
uuid: "21c659b9cfe24c87a2fe9706a2c0e254"
format_stamp: "Formatted at 2026-06-20 18:39:05 on dist-test-slave-d645"
server_key: "e3a6049f03d87e31b646183014021f6b"
server_key_iv: "c67f120301731522179b622269243275"
server_key_version: "encryptionkey@0"
I20260620 18:39:05.242511 32356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:39:05.274459 32356 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:39:05.278707 32356 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:39:05.279018 32356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:39:05.279986 32356 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:39:05.281476 32356 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:39:05.281556 32356 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:05.281682 32356 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:39:05.281741 32356 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:05.314476 32356 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.1:46607
I20260620 18:39:05.314515 32483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.1:46607 every 8 connection(s)
I20260620 18:39:05.315739 32356 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/ts-0/data/info.pb
I20260620 18:39:05.320362 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 32356
I20260620 18:39:05.320535 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/ts-0/wal/instance
I20260620 18:39:05.321121 24012 external_mini_cluster.cc:1428] Setting key c98c2eb529f2541b9c6c321a3e283541
Jun 20 18:39:05 dist-test-slave-d645 krb5kdc[32250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980745, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.1@KRBTEST.COM for kudu/127.23.115.62@KRBTEST.COM
I20260620 18:39:05.336251 32486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:05.318223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.1:59855 (local address 127.23.115.62:43103)
0620 18:39:05.318679 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:05.318684 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:05.319578 (+ 894us) server_negotiation.cc:407] Connection header received
0620 18:39:05.321371 (+ 1793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:05.321378 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:05.321470 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:05.321584 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:05.323572 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:05.324614 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:05.326264 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:05.326824 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:05.326980 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:05.330700 (+ 3720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:05.330707 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:05.330712 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:05.330754 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:05.333168 (+ 2414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:05.333968 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:05.333972 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:05.333974 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:05.334088 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:05.334544 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:05.334548 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:05.334550 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:05.334804 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:05.334922 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:05.335781 (+ 859us) server_negotiation.cc:299] Negotiation successful
0620 18:39:05.335991 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":241,"threads_started":1}
I20260620 18:39:05.337497 32487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:05.318734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:43103 (local address 127.23.115.1:59855)
0620 18:39:05.319431 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:05.319469 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:05.321092 (+ 1623us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:05.321862 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:05.321874 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:05.322601 (+ 727us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:05.323382 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:05.323402 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:05.324824 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:05.324829 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:05.326118 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:05.326133 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:05.327102 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:05.327111 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:05.327323 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:05.327975 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:05.328002 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:05.330439 (+ 2437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:39:05.333296 (+ 2857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:05.333305 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:05.333321 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:05.333849 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:05.334180 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:05.334185 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:05.334188 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:05.334422 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:05.334923 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:39:05.334960 (+ 37us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:39:05.335377 (+ 417us) client_negotiation.cc:769] Sending connection context
0620 18:39:05.335646 (+ 269us) client_negotiation.cc:240] Negotiation successful
0620 18:39:05.336163 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":304,"threads_started":1}
I20260620 18:39:05.339051 32484 heartbeater.cc:347] Connected to a master server at 127.23.115.62:43103
I20260620 18:39:05.339406 32484 heartbeater.cc:464] Registering TS with master...
I20260620 18:39:05.340276 32484 heartbeater.cc:511] Master 127.23.115.62:43103 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.23.115.2@KRBTEST.COM; defaulting to no policy
I20260620 18:39:05.342520 32295 ts_manager.cc:194] Registered new tserver with Master: 21c659b9cfe24c87a2fe9706a2c0e254 (127.23.115.1:46607)
I20260620 18:39:05.344125 32295 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.115.1@KRBTEST.COM'} at 127.23.115.1:59855
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.115.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.115.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.2.keytab.
Entry for principal kudu/127.23.115.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.2.keytab.
I20260620 18:39:05.369774 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.115.2:0
--local_ip_for_outbound_sockets=127.23.115.2
--webserver_interface=127.23.115.2
--webserver_port=0
--tserver_master_addrs=127.23.115.62:43103
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.2.keytab
--principal=kudu/127.23.115.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:38503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:39:05.679877 32490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:39:05.680143 32490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:39:05.680236 32490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:39:05.687312 32490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:39:05.687434 32490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:39:05.687544 32490 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.115.2
I20260620 18:39:05.697094 32490 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:38503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.2.keytab
--principal=kudu/127.23.115.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.115.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.115.2
--webserver_port=0
--tserver_master_addrs=127.23.115.62:43103
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.115.2
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:39:05.698309 32490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:39:05.699555 32490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:39:05.713111 32496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:05.713361 32498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:05.713588 32495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:05.714674 32490 server_base.cc:1034] running on GCE node
I20260620 18:39:05.715386 32490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:39:05.716511 32490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:39:05.717723 32490 hybrid_clock.cc:648] HybridClock initialized: now 1781980745717665 us; error 53 us; skew 500 ppm
I20260620 18:39:05.718066 32490 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:39:05 dist-test-slave-d645 krb5kdc[32250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980745, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:39:05.721756 32490 init.cc:377] Logged in from keytab as kudu/127.23.115.2@KRBTEST.COM (short username kudu)
I20260620 18:39:05.723413 32490 webserver.cc:460] Webserver started at http://127.23.115.2:46171/ using document root <none> and password file <none>
I20260620 18:39:05.724221 32490 fs_manager.cc:359] Metadata directory not provided
I20260620 18:39:05.724335 32490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:39:05.724632 32490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:39:05.727978 32490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/ts-1/data/instance:
uuid: "f3da13a2c27741d4a3c31b8ecb3bc3e3"
format_stamp: "Formatted at 2026-06-20 18:39:05 on dist-test-slave-d645"
server_key: "9ada927f35c602afaac482569f599d9b"
server_key_iv: "69956f56a912fc9780563ade599a362f"
server_key_version: "encryptionkey@0"
I20260620 18:39:05.728667 32490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/ts-1/wal/instance:
uuid: "f3da13a2c27741d4a3c31b8ecb3bc3e3"
format_stamp: "Formatted at 2026-06-20 18:39:05 on dist-test-slave-d645"
server_key: "9ada927f35c602afaac482569f599d9b"
server_key_iv: "69956f56a912fc9780563ade599a362f"
server_key_version: "encryptionkey@0"
I20260620 18:39:05.733556 32490 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:39:05.736837 32505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:05.738027 32490 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 18:39:05.738162 32490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/ts-1/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/ts-1/wal
uuid: "f3da13a2c27741d4a3c31b8ecb3bc3e3"
format_stamp: "Formatted at 2026-06-20 18:39:05 on dist-test-slave-d645"
server_key: "9ada927f35c602afaac482569f599d9b"
server_key_iv: "69956f56a912fc9780563ade599a362f"
server_key_version: "encryptionkey@0"
I20260620 18:39:05.738343 32490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:39:05.764482 32490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:39:05.768615 32490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:39:05.768910 32490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:39:05.769879 32490 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:39:05.771194 32490 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:39:05.771270 32490 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:05.771387 32490 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:39:05.771435 32490 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:05.801383 32490 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.2:45615
I20260620 18:39:05.801460 32617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.2:45615 every 8 connection(s)
I20260620 18:39:05.802577 32490 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/ts-1/data/info.pb
I20260620 18:39:05.810544 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 32490
I20260620 18:39:05.810716 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/ts-1/wal/instance
I20260620 18:39:05.811194 24012 external_mini_cluster.cc:1428] Setting key b0f0b8551fec288580eea87cb573b7b1
Jun 20 18:39:05 dist-test-slave-d645 krb5kdc[32250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980745, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.2@KRBTEST.COM for kudu/127.23.115.62@KRBTEST.COM
I20260620 18:39:05.822448 32486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:05.805084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.2:49159 (local address 127.23.115.62:43103)
0620 18:39:05.805254 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:05.805260 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:05.806394 (+ 1134us) server_negotiation.cc:407] Connection header received
0620 18:39:05.808202 (+ 1808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:05.808208 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:05.808303 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:05.808393 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:05.809823 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:05.810537 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:05.812257 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:05.812715 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:05.812825 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:05.816189 (+ 3364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:05.816195 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:05.816199 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:05.816235 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:05.819099 (+ 2864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:05.820103 (+ 1004us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:05.820107 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:05.820110 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:05.820252 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:05.820744 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:05.820748 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:05.820750 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:05.821017 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:05.821155 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:05.821850 (+ 695us) server_negotiation.cc:299] Negotiation successful
0620 18:39:05.822083 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 18:39:05.823467 32620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:05.805563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:43103 (local address 127.23.115.2:49159)
0620 18:39:05.806229 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:05.806272 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:05.807940 (+ 1668us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:05.808621 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:05.808629 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:05.809138 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:05.809664 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:05.809681 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:05.810679 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:05.810684 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:05.812139 (+ 1455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:05.812150 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:05.812971 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:05.812980 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:05.813205 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:05.813870 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:05.813897 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:05.815966 (+ 2069us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:39:05.819256 (+ 3290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:05.819263 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:05.819276 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:05.819924 (+ 648us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:05.820364 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:05.820369 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:05.820372 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:05.820616 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:05.821150 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:39:05.821158 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:39:05.821486 (+ 328us) client_negotiation.cc:769] Sending connection context
0620 18:39:05.821715 (+ 229us) client_negotiation.cc:240] Negotiation successful
0620 18:39:05.822244 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":260,"threads_started":1}
I20260620 18:39:05.824877 32618 heartbeater.cc:347] Connected to a master server at 127.23.115.62:43103
I20260620 18:39:05.825158 32618 heartbeater.cc:464] Registering TS with master...
I20260620 18:39:05.825845 32618 heartbeater.cc:511] Master 127.23.115.62:43103 requested a full tablet report, sending...
I20260620 18:39:05.827280 32295 ts_manager.cc:194] Registered new tserver with Master: f3da13a2c27741d4a3c31b8ecb3bc3e3 (127.23.115.2:45615)
I20260620 18:39:05.828398 32295 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.115.2@KRBTEST.COM'} at 127.23.115.2:49159
WARNING: no policy specified for kudu/127.23.115.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.115.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.115.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.3.keytab.
Entry for principal kudu/127.23.115.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.3.keytab.
I20260620 18:39:05.858003 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.115.3:0
--local_ip_for_outbound_sockets=127.23.115.3
--webserver_interface=127.23.115.3
--webserver_port=0
--tserver_master_addrs=127.23.115.62:43103
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.3.keytab
--principal=kudu/127.23.115.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:38503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:39:06.168709 32623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:39:06.169027 32623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:39:06.169134 32623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:39:06.176537 32623 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:39:06.176656 32623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:39:06.176772 32623 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.115.3
I20260620 18:39:06.187094 32623 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:38503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.3.keytab
--principal=kudu/127.23.115.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.115.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.115.3
--webserver_port=0
--tserver_master_addrs=127.23.115.62:43103
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.115.3
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:39:06.188392 32623 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:39:06.189658 32623 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:39:06.202647 32628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:06.202600 32629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:06.202553 32631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:06.203644 32623 server_base.cc:1034] running on GCE node
I20260620 18:39:06.204237 32623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:39:06.205152 32623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:39:06.206377 32623 hybrid_clock.cc:648] HybridClock initialized: now 1781980746206339 us; error 32 us; skew 500 ppm
I20260620 18:39:06.206701 32623 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:39:06 dist-test-slave-d645 krb5kdc[32250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980746, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:39:06.209776 32623 init.cc:377] Logged in from keytab as kudu/127.23.115.3@KRBTEST.COM (short username kudu)
I20260620 18:39:06.211338 32623 webserver.cc:460] Webserver started at http://127.23.115.3:43237/ using document root <none> and password file <none>
I20260620 18:39:06.212050 32623 fs_manager.cc:359] Metadata directory not provided
I20260620 18:39:06.212167 32623 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:39:06.212460 32623 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:39:06.215339 32623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/ts-2/data/instance:
uuid: "c8338179182e4c699653cf802f732ed2"
format_stamp: "Formatted at 2026-06-20 18:39:06 on dist-test-slave-d645"
server_key: "e60664211ef8404b409a0506f069133e"
server_key_iv: "3710024fdf9dc7c2b535292ccccfb84d"
server_key_version: "encryptionkey@0"
I20260620 18:39:06.216084 32623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/ts-2/wal/instance:
uuid: "c8338179182e4c699653cf802f732ed2"
format_stamp: "Formatted at 2026-06-20 18:39:06 on dist-test-slave-d645"
server_key: "e60664211ef8404b409a0506f069133e"
server_key_iv: "3710024fdf9dc7c2b535292ccccfb84d"
server_key_version: "encryptionkey@0"
I20260620 18:39:06.221184 32623 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:39:06.224732 32638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:06.226158 32623 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 18:39:06.226361 32623 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/ts-2/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/ts-2/wal
uuid: "c8338179182e4c699653cf802f732ed2"
format_stamp: "Formatted at 2026-06-20 18:39:06 on dist-test-slave-d645"
server_key: "e60664211ef8404b409a0506f069133e"
server_key_iv: "3710024fdf9dc7c2b535292ccccfb84d"
server_key_version: "encryptionkey@0"
I20260620 18:39:06.226553 32623 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:39:06.261262 32623 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:39:06.265978 32623 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:39:06.266252 32623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:39:06.267221 32623 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:39:06.268622 32623 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:39:06.268698 32623 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:06.268831 32623 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:39:06.268883 32623 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:06.299988 32623 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.3:39625
I20260620 18:39:06.300041 32750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.3:39625 every 8 connection(s)
I20260620 18:39:06.301342 32623 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/ts-2/data/info.pb
I20260620 18:39:06.308487 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 32623
I20260620 18:39:06.308660 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/ts-2/wal/instance
I20260620 18:39:06.309211 24012 external_mini_cluster.cc:1428] Setting key cc2c4e0b34d26a616ab02f2cda433914
Jun 20 18:39:06 dist-test-slave-d645 krb5kdc[32250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980746, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.3@KRBTEST.COM for kudu/127.23.115.62@KRBTEST.COM
I20260620 18:39:06.323086 32486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:06.304065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.3:34387 (local address 127.23.115.62:43103)
0620 18:39:06.304304 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:06.304309 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:06.305702 (+ 1393us) server_negotiation.cc:407] Connection header received
0620 18:39:06.307639 (+ 1937us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:06.307645 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:06.307731 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:06.307816 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:06.309856 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:06.310838 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:06.312663 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:06.313085 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:06.313193 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:06.316993 (+ 3800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:06.316999 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:06.317002 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:06.317038 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:06.319837 (+ 2799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:06.320551 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:06.320555 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:06.320557 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:06.320679 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:06.321090 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:06.321093 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:06.321095 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:06.321378 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:06.321513 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:06.322384 (+ 871us) server_negotiation.cc:299] Negotiation successful
0620 18:39:06.322642 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260620 18:39:06.324069 32753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:06.304752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:43103 (local address 127.23.115.3:34387)
0620 18:39:06.305520 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:06.305558 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:06.307347 (+ 1789us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:06.308102 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:06.308112 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:06.308847 (+ 735us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:06.309621 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:06.309647 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:06.311034 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:06.311040 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:06.312507 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:06.312522 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:06.313380 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:06.313386 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:06.313608 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:06.314357 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:06.314400 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:06.316756 (+ 2356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:39:06.319983 (+ 3227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:06.319993 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:06.320008 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:06.320420 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:06.320785 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:06.320790 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:06.320801 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:06.320977 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:06.321507 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:39:06.321515 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:39:06.321918 (+ 403us) client_negotiation.cc:769] Sending connection context
0620 18:39:06.322195 (+ 277us) client_negotiation.cc:240] Negotiation successful
0620 18:39:06.322762 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":327,"threads_started":1}
I20260620 18:39:06.325351 32751 heartbeater.cc:347] Connected to a master server at 127.23.115.62:43103
I20260620 18:39:06.325656 32751 heartbeater.cc:464] Registering TS with master...
I20260620 18:39:06.326330 32751 heartbeater.cc:511] Master 127.23.115.62:43103 requested a full tablet report, sending...
I20260620 18:39:06.327991 32294 ts_manager.cc:194] Registered new tserver with Master: c8338179182e4c699653cf802f732ed2 (127.23.115.3:39625)
I20260620 18:39:06.328828 32294 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.115.3@KRBTEST.COM'} at 127.23.115.3:34387
I20260620 18:39:06.337396 24012 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:39:06.347165 32484 heartbeater.cc:503] Master 127.23.115.62:43103 was elected leader, sending a full tablet report...
WARNING: no policy specified for oryx/127.23.115.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.115.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.115.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/krb5kdc/oryx_127.23.115.1.keytab.
Entry for principal oryx/127.23.115.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/krb5kdc/oryx_127.23.115.1.keytab.
WARNING: no policy specified for oryx/127.23.115.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.115.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.115.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/krb5kdc/oryx_127.23.115.2.keytab.
Entry for principal oryx/127.23.115.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/krb5kdc/oryx_127.23.115.2.keytab.
WARNING: no policy specified for oryx/127.23.115.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.115.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.115.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/krb5kdc/oryx_127.23.115.3.keytab.
Entry for principal oryx/127.23.115.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/krb5kdc/oryx_127.23.115.3.keytab.
I20260620 18:39:06.466466 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 32356
Jun 20 18:39:06 dist-test-slave-d645 krb5kdc[32250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980744, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.115.1@KRBTEST.COM
I20260620 18:39:06.483172 32761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:06.467696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52928 (local address 127.23.115.1:46607)
0620 18:39:06.468308 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:06.468315 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:06.468344 (+ 29us) server_negotiation.cc:407] Connection header received
0620 18:39:06.468410 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:06.468416 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:06.468643 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:06.468770 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:06.470125 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:06.471155 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:06.472427 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:06.472985 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:06.473099 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:06.477620 (+ 4521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:06.477633 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:06.477647 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:06.477687 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:06.480363 (+ 2676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:06.481068 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:06.481076 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:06.481079 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:06.481211 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:06.481595 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:06.481599 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:06.481601 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:06.481997 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:06.482256 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:06.482554 (+ 298us) server_negotiation.cc:299] Negotiation successful
0620 18:39:06.482759 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":296,"threads_started":1}
I20260620 18:39:06.484371 32479 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:39:06.565907 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 32356
I20260620 18:39:06.579988 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 32490
Jun 20 18:39:06 dist-test-slave-d645 krb5kdc[32250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980744, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.115.2@KRBTEST.COM
I20260620 18:39:06.597738 32763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:06.581035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47860 (local address 127.23.115.2:45615)
0620 18:39:06.581655 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:06.581662 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:06.581688 (+ 26us) server_negotiation.cc:407] Connection header received
0620 18:39:06.581747 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:06.581753 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:06.581954 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:06.582081 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:06.583385 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:06.584417 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:06.585733 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:06.586291 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:06.586398 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:06.591216 (+ 4818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:06.591231 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:06.591247 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:06.591287 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:06.594191 (+ 2904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:06.595183 (+ 992us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:06.595191 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:06.595193 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:06.595377 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:06.595836 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:06.595840 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:06.595842 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:06.596231 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:06.596495 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:06.597057 (+ 562us) server_negotiation.cc:299] Negotiation successful
0620 18:39:06.597286 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":479,"thread_start_us":362,"threads_started":1}
I20260620 18:39:06.599032 32613 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:39:06.683072 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 32490
I20260620 18:39:06.696166 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 32623
Jun 20 18:39:06 dist-test-slave-d645 krb5kdc[32250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980744, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.115.3@KRBTEST.COM
I20260620 18:39:06.717403 32765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:06.697475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41420 (local address 127.23.115.3:39625)
0620 18:39:06.698157 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:06.698164 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:06.698199 (+ 35us) server_negotiation.cc:407] Connection header received
0620 18:39:06.698275 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:06.698285 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:06.698527 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:06.698676 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:06.700294 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:06.701590 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:06.703371 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:06.704173 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:06.704359 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:06.710821 (+ 6462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:06.710844 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:06.710868 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:06.710967 (+ 99us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:06.714362 (+ 3395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:06.715292 (+ 930us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:06.715300 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:06.715302 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:06.715446 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:06.715857 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:06.715862 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:06.715864 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:06.716264 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:06.716541 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:06.716843 (+ 302us) server_negotiation.cc:299] Negotiation successful
0620 18:39:06.717063 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":339,"threads_started":1}
I20260620 18:39:06.718649 32746 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:39:06.800473 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 32623
I20260620 18:39:06.812918 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 32263
I20260620 18:39:06.814155 32325 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:39:06.915302 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 32263
I20260620 18:39:06.933921 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.115.62:43103
--webserver_interface=127.23.115.62
--webserver_port=37549
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab
--principal=kudu/127.23.115.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:38503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.115.62:43103
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:39:07.254163 300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:39:07.254482 300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:39:07.254576 300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:39:07.262176 300 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:39:07.262305 300 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:39:07.262352 300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:39:07.262421 300 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:39:07.262466 300 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:39:07.272598 300 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:38503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.115.62:43103
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab
--principal=kudu/127.23.115.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.115.62:43103
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.115.62
--webserver_port=37549
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:39:07.273916 300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:39:07.275209 300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:39:07.288968 305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:07.289800 308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:07.290336 306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:07.291811 300 server_base.cc:1034] running on GCE node
I20260620 18:39:07.292636 300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:39:07.294149 300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:39:07.295382 300 hybrid_clock.cc:648] HybridClock initialized: now 1781980747295335 us; error 36 us; skew 500 ppm
I20260620 18:39:07.295734 300 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:39:07 dist-test-slave-d645 krb5kdc[32250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980747, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:39:07.298686 300 init.cc:377] Logged in from keytab as kudu/127.23.115.62@KRBTEST.COM (short username kudu)
I20260620 18:39:07.300238 300 webserver.cc:460] Webserver started at http://127.23.115.62:37549/ using document root <none> and password file <none>
I20260620 18:39:07.300935 300 fs_manager.cc:359] Metadata directory not provided
I20260620 18:39:07.301051 300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:39:07.306516 300 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260620 18:39:07.310319 315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:07.311846 300 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 18:39:07.312011 300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/master-0/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/master-0/wal
uuid: "3cd2627faa2e4667b6c5f8d7e6408827"
format_stamp: "Formatted at 2026-06-20 18:39:04 on dist-test-slave-d645"
server_key: "b5a492cd09578221b07adeed77ae87eb"
server_key_iv: "a59ad8582878518b1d1d65d8b4b4cb5b"
server_key_version: "encryptionkey@0"
I20260620 18:39:07.312978 300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:39:07.335923 300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:39:07.340058 300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:39:07.340277 300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:39:07.357733 300 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.62:43103
I20260620 18:39:07.357781 366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.62:43103 every 8 connection(s)
I20260620 18:39:07.359249 300 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
I20260620 18:39:07.365466 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 300
I20260620 18:39:07.366246 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.115.1:46607
--local_ip_for_outbound_sockets=127.23.115.1
--tserver_master_addrs=127.23.115.62:43103
--webserver_port=34187
--webserver_interface=127.23.115.1
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.1.keytab
--principal=kudu/127.23.115.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:38503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.23.115.1
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/krb5kdc/oryx_127.23.115.1.keytab with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:39:07.366508 367 sys_catalog.cc:263] Verifying existing consensus state
I20260620 18:39:07.369833 367 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3cd2627faa2e4667b6c5f8d7e6408827: Bootstrap starting.
I20260620 18:39:07.394804 367 log.cc:824] T 00000000000000000000000000000000 P 3cd2627faa2e4667b6c5f8d7e6408827: Log is configured to *not* fsync() on all Append() calls
I20260620 18:39:07.405967 367 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3cd2627faa2e4667b6c5f8d7e6408827: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 18:39:07.406565 367 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3cd2627faa2e4667b6c5f8d7e6408827: Bootstrap complete.
I20260620 18:39:07.419643 367 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3cd2627faa2e4667b6c5f8d7e6408827 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cd2627faa2e4667b6c5f8d7e6408827" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 43103 } }
I20260620 18:39:07.420812 367 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3cd2627faa2e4667b6c5f8d7e6408827 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cd2627faa2e4667b6c5f8d7e6408827, State: Initialized, Role: FOLLOWER
I20260620 18:39:07.421584 367 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3cd2627faa2e4667b6c5f8d7e6408827 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cd2627faa2e4667b6c5f8d7e6408827" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 43103 } }
I20260620 18:39:07.421876 367 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3cd2627faa2e4667b6c5f8d7e6408827 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:39:07.422004 367 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3cd2627faa2e4667b6c5f8d7e6408827 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:39:07.422176 367 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3cd2627faa2e4667b6c5f8d7e6408827 [term 1 FOLLOWER]: Advancing to term 2
I20260620 18:39:07.427745 367 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3cd2627faa2e4667b6c5f8d7e6408827 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cd2627faa2e4667b6c5f8d7e6408827" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 43103 } }
I20260620 18:39:07.428412 367 leader_election.cc:302] T 00000000000000000000000000000000 P 3cd2627faa2e4667b6c5f8d7e6408827 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3cd2627faa2e4667b6c5f8d7e6408827; no voters:
I20260620 18:39:07.429097 367 leader_election.cc:288] T 00000000000000000000000000000000 P 3cd2627faa2e4667b6c5f8d7e6408827 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 18:39:07.429236 371 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3cd2627faa2e4667b6c5f8d7e6408827 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 18:39:07.430361 371 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3cd2627faa2e4667b6c5f8d7e6408827 [term 2 LEADER]: Becoming Leader. State: Replica: 3cd2627faa2e4667b6c5f8d7e6408827, State: Running, Role: LEADER
I20260620 18:39:07.430903 371 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3cd2627faa2e4667b6c5f8d7e6408827 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cd2627faa2e4667b6c5f8d7e6408827" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 43103 } }
I20260620 18:39:07.431406 367 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3cd2627faa2e4667b6c5f8d7e6408827 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:39:07.433279 372 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cd2627faa2e4667b6c5f8d7e6408827 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "3cd2627faa2e4667b6c5f8d7e6408827" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cd2627faa2e4667b6c5f8d7e6408827" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 43103 } } }
I20260620 18:39:07.433272 373 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cd2627faa2e4667b6c5f8d7e6408827 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3cd2627faa2e4667b6c5f8d7e6408827. Latest consensus state: current_term: 2 leader_uuid: "3cd2627faa2e4667b6c5f8d7e6408827" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cd2627faa2e4667b6c5f8d7e6408827" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 43103 } } }
I20260620 18:39:07.433688 373 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cd2627faa2e4667b6c5f8d7e6408827 [sys.catalog]: This master's current role is: LEADER
I20260620 18:39:07.433686 372 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cd2627faa2e4667b6c5f8d7e6408827 [sys.catalog]: This master's current role is: LEADER
I20260620 18:39:07.437973 377 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:39:07.443475 377 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:39:07.444888 377 catalog_manager.cc:1259] Loaded cluster ID: 29233afe22974cbf97c8c083df9c2306
I20260620 18:39:07.445158 377 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:39:07.449092 377 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:39:07.450524 377 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3cd2627faa2e4667b6c5f8d7e6408827: Loaded TSK: 0
I20260620 18:39:07.452019 377 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 18:39:07.711620 369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:39:07.711887 369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:39:07.711992 369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:39:07.719404 369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:39:07.719516 369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:39:07.719656 369 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.115.1
I20260620 18:39:07.729445 369 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:38503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/krb5kdc/oryx_127.23.115.1.keytab
--principal=oryx/127.23.115.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.115.1:46607
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.115.1
--webserver_port=34187
--tserver_master_addrs=127.23.115.62:43103
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.115.1
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:39:07.730755 369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:39:07.732121 369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:39:07.746796 394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:07.747071 393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:07.746999 396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:07.747633 369 server_base.cc:1034] running on GCE node
I20260620 18:39:07.748239 369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:39:07.749204 369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:39:07.750447 369 hybrid_clock.cc:648] HybridClock initialized: now 1781980747750347 us; error 61 us; skew 500 ppm
I20260620 18:39:07.750746 369 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:39:07 dist-test-slave-d645 krb5kdc[32250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980747, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.115.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:39:07.754858 369 init.cc:377] Logged in from keytab as oryx/127.23.115.1@KRBTEST.COM (short username oryx)
I20260620 18:39:07.756712 369 webserver.cc:460] Webserver started at http://127.23.115.1:34187/ using document root <none> and password file <none>
I20260620 18:39:07.757501 369 fs_manager.cc:359] Metadata directory not provided
I20260620 18:39:07.757642 369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:39:07.763939 369 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 18:39:07.766808 403 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:07.768307 369 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 18:39:07.768476 369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/ts-0/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/ts-0/wal
uuid: "21c659b9cfe24c87a2fe9706a2c0e254"
format_stamp: "Formatted at 2026-06-20 18:39:05 on dist-test-slave-d645"
server_key: "e3a6049f03d87e31b646183014021f6b"
server_key_iv: "c67f120301731522179b622269243275"
server_key_version: "encryptionkey@0"
I20260620 18:39:07.769447 369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:39:07.791435 369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:39:07.795737 369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:39:07.795989 369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:39:07.796947 369 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:39:07.798264 369 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:39:07.798355 369 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:07.798465 369 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:39:07.798542 369 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:07.828751 369 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.1:46607
I20260620 18:39:07.828840 515 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.1:46607 every 8 connection(s)
I20260620 18:39:07.830125 369 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/ts-0/data/info.pb
I20260620 18:39:07.831475 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 369
I20260620 18:39:07.832201 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.115.2:45615
--local_ip_for_outbound_sockets=127.23.115.2
--tserver_master_addrs=127.23.115.62:43103
--webserver_port=46171
--webserver_interface=127.23.115.2
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.2.keytab
--principal=kudu/127.23.115.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:38503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.23.115.2
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/krb5kdc/oryx_127.23.115.2.keytab with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 18:39:07 dist-test-slave-d645 krb5kdc[32250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.115.1@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:39:07 dist-test-slave-d645 krb5kdc[32250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.115.1@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 18:39:08.163105 520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:39:08.163429 520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:39:08.163563 520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:39:08.171273 520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:39:08.171420 520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:39:08.171582 520 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.115.2
I20260620 18:39:08.181815 520 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:38503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/krb5kdc/oryx_127.23.115.2.keytab
--principal=oryx/127.23.115.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.115.2:45615
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.115.2
--webserver_port=46171
--tserver_master_addrs=127.23.115.62:43103
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.115.2
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:39:08.183202 520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:39:08.184756 520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:39:08.200773 526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:08.200722 527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:08.200712 529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:08.201682 520 server_base.cc:1034] running on GCE node
I20260620 18:39:08.202310 520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:39:08.203362 520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:39:08.204588 520 hybrid_clock.cc:648] HybridClock initialized: now 1781980748204527 us; error 51 us; skew 500 ppm
I20260620 18:39:08.204918 520 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:39:08 dist-test-slave-d645 krb5kdc[32250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980748, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.115.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:39:08.208335 520 init.cc:377] Logged in from keytab as oryx/127.23.115.2@KRBTEST.COM (short username oryx)
I20260620 18:39:08.210680 520 webserver.cc:460] Webserver started at http://127.23.115.2:46171/ using document root <none> and password file <none>
I20260620 18:39:08.211454 520 fs_manager.cc:359] Metadata directory not provided
I20260620 18:39:08.211581 520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:39:08.217321 520 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 18:39:08.220059 536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:08.221415 520 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:39:08.221566 520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/ts-1/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/ts-1/wal
uuid: "f3da13a2c27741d4a3c31b8ecb3bc3e3"
format_stamp: "Formatted at 2026-06-20 18:39:05 on dist-test-slave-d645"
server_key: "9ada927f35c602afaac482569f599d9b"
server_key_iv: "69956f56a912fc9780563ade599a362f"
server_key_version: "encryptionkey@0"
I20260620 18:39:08.222518 520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:39:08.266145 520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:39:08.270406 520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:39:08.270673 520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:39:08.271744 520 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:39:08.273140 520 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:39:08.273239 520 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:08.273345 520 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:39:08.273428 520 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:08.308313 520 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.2:45615
I20260620 18:39:08.308344 648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.2:45615 every 8 connection(s)
I20260620 18:39:08.309715 520 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/ts-1/data/info.pb
I20260620 18:39:08.318063 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 520
I20260620 18:39:08.318882 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.115.3:39625
--local_ip_for_outbound_sockets=127.23.115.3
--tserver_master_addrs=127.23.115.62:43103
--webserver_port=43237
--webserver_interface=127.23.115.3
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.3.keytab
--principal=kudu/127.23.115.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:38503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.23.115.3
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/krb5kdc/oryx_127.23.115.3.keytab with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 18:39:08 dist-test-slave-d645 krb5kdc[32250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.115.2@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:39:08 dist-test-slave-d645 krb5kdc[32250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.115.2@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 18:39:08.851450 653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:39:08.851866 653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:39:08.852110 653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:39:08.863736 653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:39:08.863881 653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:39:08.864125 653 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.115.3
I20260620 18:39:08.880141 653 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:38503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/krb5kdc/oryx_127.23.115.3.keytab
--principal=oryx/127.23.115.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.115.3:39625
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.115.3
--webserver_port=43237
--tserver_master_addrs=127.23.115.62:43103
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.115.3
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:39:08.881898 653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:39:08.883564 653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:39:08.903488 659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:08.907326 660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:08.909248 662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:08.910164 653 server_base.cc:1034] running on GCE node
I20260620 18:39:08.911103 653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:39:08.912523 653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:39:08.913829 653 hybrid_clock.cc:648] HybridClock initialized: now 1781980748913769 us; error 40 us; skew 500 ppm
I20260620 18:39:08.914289 653 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:39:08 dist-test-slave-d645 krb5kdc[32250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980748, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.115.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:39:08.919134 653 init.cc:377] Logged in from keytab as oryx/127.23.115.3@KRBTEST.COM (short username oryx)
I20260620 18:39:08.921432 653 webserver.cc:460] Webserver started at http://127.23.115.3:43237/ using document root <none> and password file <none>
I20260620 18:39:08.922570 653 fs_manager.cc:359] Metadata directory not provided
I20260620 18:39:08.922745 653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:39:08.931414 653 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.007s sys 0.001s
I20260620 18:39:08.935626 669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:08.937506 653 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260620 18:39:08.937786 653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/ts-2/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/ts-2/wal
uuid: "c8338179182e4c699653cf802f732ed2"
format_stamp: "Formatted at 2026-06-20 18:39:06 on dist-test-slave-d645"
server_key: "e60664211ef8404b409a0506f069133e"
server_key_iv: "3710024fdf9dc7c2b535292ccccfb84d"
server_key_version: "encryptionkey@0"
I20260620 18:39:08.939512 653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20260620 18:39:08.956144 519 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.110s user 0.003s sys 0.007s
I20260620 18:39:08.957991 519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:07.833546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:43103 (local address 127.23.115.1:58959)
0620 18:39:07.834960 (+ 1414us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:07.835018 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:07.837163 (+ 2145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:07.838623 (+ 1460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:07.838634 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:07.839454 (+ 820us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:07.840013 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:07.840038 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:07.842264 (+ 2226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:07.842269 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:07.843811 (+ 1542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:07.843825 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:07.845090 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:07.845096 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:07.845303 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:07.845947 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:07.845973 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:08.956888 (+1110915us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.115.62:43103: Server oryx/127.23.115.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":955,"thread_start_us":683,"threads_started":1}
W20260620 18:39:08.957984 518 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:07.833533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.1:58959 (local address 127.23.115.62:43103)
0620 18:39:07.834275 (+ 742us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:07.834304 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:07.835146 (+ 842us) server_negotiation.cc:407] Connection header received
0620 18:39:07.837765 (+ 2619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:07.837780 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:07.838122 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:07.838424 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:07.840252 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:07.842084 (+ 1832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:07.843949 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:07.844851 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:07.845015 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:08.957275 (+1112260us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.115.1:58959: BlockingRecv error: recv got EOF from 127.23.115.1:58959 (error 108)
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":251,"threads_started":1}
W20260620 18:39:08.959710 516 heartbeater.cc:644] Failed to heartbeat to 127.23.115.62:43103 (0 consecutive failures): Not authorized: Failed to ping master at 127.23.115.62:43103: Client connection negotiation failed: client connection to 127.23.115.62:43103: Server oryx/127.23.115.62@KRBTEST.COM not found in Kerberos database
I20260620 18:39:08.959679 653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:39:08.966169 653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:39:08.966526 653 kserver.cc:163] Server-wide thread pool size limit: 3276
Jun 20 18:39:08 dist-test-slave-d645 krb5kdc[32250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.115.1@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:39:08 dist-test-slave-d645 krb5kdc[32250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.115.1@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:39:08.967922 653 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:39:08.969218 519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:08.960410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:43103 (local address 127.23.115.1:55261)
0620 18:39:08.960599 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:08.960617 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:08.960733 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:08.961381 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:08.961386 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:08.961752 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:08.961909 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:08.961918 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:08.963099 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:08.963105 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:08.964174 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:08.964188 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:08.964871 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:08.964877 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:08.965041 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:08.965515 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:08.965532 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:08.968936 (+ 3404us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.115.62:43103: Server oryx/127.23.115.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":64}
I20260620 18:39:08.970270 653 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:39:08.970384 653 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:08.970489 653 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:39:08.970574 653 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260620 18:39:08.970780 518 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:08.960959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.1:55261 (local address 127.23.115.62:43103)
0620 18:39:08.961109 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:08.961113 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:08.961129 (+ 16us) server_negotiation.cc:407] Connection header received
0620 18:39:08.961180 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:08.961183 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:08.961238 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:08.961327 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:08.962051 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:08.962905 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:08.964307 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:08.964706 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:08.964814 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:08.969165 (+ 4351us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.115.1:55261: BlockingRecv error: recv got EOF from 127.23.115.1:55261 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 20 18:39:08 dist-test-slave-d645 krb5kdc[32250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.115.1@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:39:08 dist-test-slave-d645 krb5kdc[32250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.115.1@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:39:08.983353 519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:08.970024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:43103 (local address 127.23.115.1:55147)
0620 18:39:08.970246 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:08.970263 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:08.970366 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:08.971586 (+ 1220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:08.971592 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:08.972020 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:08.972343 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:08.972364 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:08.974194 (+ 1830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:08.974201 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:08.975803 (+ 1602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:08.975821 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:08.976906 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:08.976913 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:08.977147 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:08.977859 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:08.977883 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:08.983029 (+ 5146us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.115.62:43103: Server oryx/127.23.115.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127}
W20260620 18:39:08.984243 518 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:08.970161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.1:55147 (local address 127.23.115.62:43103)
0620 18:39:08.971222 (+ 1061us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:08.971228 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:08.971247 (+ 19us) server_negotiation.cc:407] Connection header received
0620 18:39:08.971307 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:08.971313 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:08.971414 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:08.972532 (+ 1118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:08.972690 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:08.974014 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:08.976028 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:08.976722 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:08.982035 (+ 5313us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:08.983946 (+ 1911us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.115.1:55147: BlockingRecv error: recv got EOF from 127.23.115.1:55147 (error 108)
Metrics: {"server-negotiator.queue_time_us":950,"threads_started":1}
W20260620 18:39:08.984851 516 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 18:39:09.022555 653 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.3:39625
I20260620 18:39:09.022607 782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.3:39625 every 8 connection(s)
I20260620 18:39:09.024724 653 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980676253068-24012-0/minicluster-data/ts-2/data/info.pb
I20260620 18:39:09.031193 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 653
Jun 20 18:39:09 dist-test-slave-d645 krb5kdc[32250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.115.3@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:39:09 dist-test-slave-d645 krb5kdc[32250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.115.3@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:39:09.047201 518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:09.032352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54054 (local address 127.23.115.62:43103)
0620 18:39:09.032560 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:09.032565 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:09.033135 (+ 570us) server_negotiation.cc:407] Connection header received
0620 18:39:09.033290 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:09.033297 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:09.033395 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:09.033508 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:09.035233 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:09.035965 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:09.037191 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:09.037620 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:09.037718 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:09.040555 (+ 2837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:09.040563 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:09.040581 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:09.040623 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:09.043537 (+ 2914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:09.044170 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:09.044182 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:09.044193 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:09.044441 (+ 248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:09.044812 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:09.044818 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:09.044822 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:09.045604 (+ 782us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:09.045949 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:09.046566 (+ 617us) server_negotiation.cc:299] Negotiation successful
0620 18:39:09.046912 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
W20260620 18:39:09.353030 652 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.031s user 0.000s sys 0.012s
W20260620 18:39:09.354306 651 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:08.312264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.2:43965 (local address 127.23.115.62:43103)
0620 18:39:08.312761 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:08.312769 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:08.313723 (+ 954us) server_negotiation.cc:407] Connection header received
0620 18:39:08.315571 (+ 1848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:08.315579 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:08.315673 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:08.315829 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:08.318330 (+ 2501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:08.319185 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:08.320735 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:08.321135 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:08.321243 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:09.354006 (+1032763us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.115.2:43965: BlockingRecv error: recv got EOF from 127.23.115.2:43965 (error 108)
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":218,"threads_started":1}
I20260620 18:39:09.354832 652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:08.312804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:43103 (local address 127.23.115.2:43965)
0620 18:39:08.313569 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:08.313612 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:08.315329 (+ 1717us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:08.316292 (+ 963us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:08.316307 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:08.317130 (+ 823us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:08.318083 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:08.318112 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:08.319366 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:08.319370 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:08.320592 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:08.320605 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:08.321348 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:08.321356 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:08.321596 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:08.322230 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:08.322252 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:09.353772 (+1031520us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.115.62:43103: Server oryx/127.23.115.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":297,"threads_started":1}
W20260620 18:39:09.355749 649 heartbeater.cc:644] Failed to heartbeat to 127.23.115.62:43103 (0 consecutive failures): Not authorized: Failed to ping master at 127.23.115.62:43103: Client connection negotiation failed: client connection to 127.23.115.62:43103: Server oryx/127.23.115.62@KRBTEST.COM not found in Kerberos database
Jun 20 18:39:09 dist-test-slave-d645 krb5kdc[32250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.115.2@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:39:09 dist-test-slave-d645 krb5kdc[32250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.115.2@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:39:09.368746 652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:09.356825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:43103 (local address 127.23.115.2:52887)
0620 18:39:09.357290 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:09.357316 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:09.357554 (+ 238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:09.358182 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:09.358188 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:09.358742 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:09.359016 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:09.359031 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:09.360569 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:09.360576 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:09.362231 (+ 1655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:09.362248 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:09.363287 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:09.363294 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:09.363491 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:09.364241 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:09.364267 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:09.368453 (+ 4186us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.115.62:43103: Server oryx/127.23.115.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":297}
W20260620 18:39:09.368911 651 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:09.356895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.2:52887 (local address 127.23.115.62:43103)
0620 18:39:09.357177 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:09.357186 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:09.357701 (+ 515us) server_negotiation.cc:407] Connection header received
0620 18:39:09.357780 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:09.357787 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:09.357872 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:09.357971 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:09.359172 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:09.360360 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:09.362406 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:09.363106 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:09.363229 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:09.368678 (+ 5449us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.115.2:52887: BlockingRecv error: recv got EOF from 127.23.115.2:52887 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Jun 20 18:39:09 dist-test-slave-d645 krb5kdc[32250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.115.2@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:39:09 dist-test-slave-d645 krb5kdc[32250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.115.2@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:39:09.379009 652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:09.369666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:43103 (local address 127.23.115.2:59939)
0620 18:39:09.370022 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:09.370039 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:09.370143 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:09.370558 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:09.370562 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:09.370909 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:09.371144 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:09.371156 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:09.372919 (+ 1763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:09.372923 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:09.374028 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:09.374040 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:09.375068 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:09.375072 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:09.375218 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:09.375763 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:09.375779 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:09.378772 (+ 2993us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.115.62:43103: Server oryx/127.23.115.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":259}
W20260620 18:39:09.379303 649 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 18:39:09.379395 651 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:09.369873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.2:59939 (local address 127.23.115.62:43103)
0620 18:39:09.370024 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:09.370030 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:09.370085 (+ 55us) server_negotiation.cc:407] Connection header received
0620 18:39:09.370310 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:09.370316 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:09.370416 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:09.370508 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:09.371310 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:09.372773 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:09.374171 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:09.374884 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:09.375054 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:09.379091 (+ 4037us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.115.2:59939: BlockingRecv error: recv got EOF from 127.23.115.2:59939 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Jun 20 18:39:09 dist-test-slave-d645 krb5kdc[32250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.115.1@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:39:09 dist-test-slave-d645 krb5kdc[32250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.115.1@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:39:10.000761 788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:09.985563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:43103 (local address 127.23.115.1:43131)
0620 18:39:09.986098 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:09.986123 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:09.986254 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:09.986609 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:09.986613 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:09.987082 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:09.987229 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:09.987238 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:09.988557 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:09.988563 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:09.990115 (+ 1552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:09.990132 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:09.990870 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:09.990876 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:09.991082 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:09.995930 (+ 4848us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:09.995957 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:10.000385 (+ 4428us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.115.62:43103: Server oryx/127.23.115.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":291,"threads_started":1}
W20260620 18:39:10.001199 789 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:09.985653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.1:43131 (local address 127.23.115.62:43103)
0620 18:39:09.986322 (+ 669us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:09.986329 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:09.986345 (+ 16us) server_negotiation.cc:407] Connection header received
0620 18:39:09.986396 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:09.986399 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:09.986466 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:09.986553 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:09.987373 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:09.988390 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:09.990284 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:09.990693 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:09.990791 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:10.000680 (+ 9889us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.115.1:43131: BlockingRecv error: recv got EOF from 127.23.115.1:43131 (error 108)
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":443,"threads_started":1}
W20260620 18:39:10.055464 785 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.014s user 0.009s sys 0.011s
W20260620 18:39:10.057063 676 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:09.028127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.3:57927 (local address 127.23.115.62:43103)
0620 18:39:09.028384 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:09.028389 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:09.029794 (+ 1405us) server_negotiation.cc:407] Connection header received
0620 18:39:09.031972 (+ 2178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:09.031977 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:09.032073 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:09.032177 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:09.034587 (+ 2410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:09.036020 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:09.039136 (+ 3116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:09.039770 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:09.039897 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:10.056852 (+1016955us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.115.3:57927: BlockingRecv error: recv got EOF from 127.23.115.3:57927 (error 108)
Metrics: {"server-negotiator.queue_time_us":120}
I20260620 18:39:10.057303 785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:09.028637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:43103 (local address 127.23.115.3:57927)
0620 18:39:09.029586 (+ 949us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:09.029627 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:09.031654 (+ 2027us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:09.032823 (+ 1169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:09.032836 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:09.033557 (+ 721us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:09.034371 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:09.034398 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:09.036229 (+ 1831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:09.036236 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:09.038954 (+ 2718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:09.038976 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:09.040021 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:09.040028 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:09.040382 (+ 354us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:09.041214 (+ 832us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:09.041245 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:10.056347 (+1015102us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.115.62:43103: Server oryx/127.23.115.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":642,"thread_start_us":383,"threads_started":1}
W20260620 18:39:10.058231 783 heartbeater.cc:644] Failed to heartbeat to 127.23.115.62:43103 (0 consecutive failures): Not authorized: Failed to ping master at 127.23.115.62:43103: Client connection negotiation failed: client connection to 127.23.115.62:43103: Server oryx/127.23.115.62@KRBTEST.COM not found in Kerberos database
Jun 20 18:39:10 dist-test-slave-d645 krb5kdc[32250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.115.3@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:39:10 dist-test-slave-d645 krb5kdc[32250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.115.3@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:39:10.067273 785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:10.058810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:43103 (local address 127.23.115.3:38245)
0620 18:39:10.059026 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:10.059047 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:10.059163 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:10.059434 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:10.059437 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:10.059764 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:10.059926 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:10.059934 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:10.060956 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:10.060961 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:10.061954 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:10.061971 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:10.062733 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:10.062741 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:10.062873 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:10.063559 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:10.063578 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:10.067034 (+ 3456us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.115.62:43103: Server oryx/127.23.115.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":56}
W20260620 18:39:10.067543 676 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:10.058960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.3:38245 (local address 127.23.115.62:43103)
0620 18:39:10.059185 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:10.059189 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:10.059205 (+ 16us) server_negotiation.cc:407] Connection header received
0620 18:39:10.059250 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:10.059254 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:10.059310 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:10.059386 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:10.060062 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:10.060808 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:10.062118 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:10.062526 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:10.062636 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:10.067266 (+ 4630us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.115.3:38245: BlockingRecv error: recv got EOF from 127.23.115.3:38245 (error 108)
Metrics: {"server-negotiator.queue_time_us":128}
Jun 20 18:39:10 dist-test-slave-d645 krb5kdc[32250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.115.3@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:39:10 dist-test-slave-d645 krb5kdc[32250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.115.3@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:39:10.076328 785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:10.067998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:43103 (local address 127.23.115.3:54033)
0620 18:39:10.068159 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:10.068179 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:10.068275 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:10.068649 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:10.068653 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:10.068998 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:10.069183 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:10.069191 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:10.070308 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:10.070311 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:10.071336 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:10.071347 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:10.072033 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:10.072037 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:10.072186 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:10.072760 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:10.072785 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:10.076122 (+ 3337us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.115.62:43103: Server oryx/127.23.115.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 18:39:10.076609 676 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:10.068052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.3:54033 (local address 127.23.115.62:43103)
0620 18:39:10.068207 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:10.068211 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:10.068226 (+ 15us) server_negotiation.cc:407] Connection header received
0620 18:39:10.068428 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:10.068433 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:10.068515 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:10.068610 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:10.069310 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:10.070037 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:10.071475 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:10.071890 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:10.072017 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:10.076358 (+ 4341us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.115.3:54033: BlockingRecv error: recv got EOF from 127.23.115.3:54033 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260620 18:39:10.076648 783 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 20 18:39:10 dist-test-slave-d645 krb5kdc[32250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.115.2@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:39:10 dist-test-slave-d645 krb5kdc[32250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.115.2@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:39:10.389271 790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:10.379858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:43103 (local address 127.23.115.2:57023)
0620 18:39:10.380368 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:10.380384 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:10.380494 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:10.380773 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:10.380776 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:10.381187 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:10.381327 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:10.381337 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:10.382797 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:10.382801 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:10.383815 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:10.383827 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:10.384968 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:10.384973 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:10.385102 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:10.385701 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:10.385719 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:10.389024 (+ 3305us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.115.62:43103: Server oryx/127.23.115.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":286,"threads_started":1}
W20260620 18:39:10.389474 676 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:10.379903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.2:57023 (local address 127.23.115.62:43103)
0620 18:39:10.380093 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:10.380098 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:10.380453 (+ 355us) server_negotiation.cc:407] Connection header received
0620 18:39:10.380587 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:10.380591 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:10.380666 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:10.380744 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:10.381608 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:10.382646 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:10.383960 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:10.384752 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:10.384868 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:10.389226 (+ 4358us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.115.2:57023: BlockingRecv error: recv got EOF from 127.23.115.2:57023 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 20 18:39:11 dist-test-slave-d645 krb5kdc[32250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.115.1@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:39:11 dist-test-slave-d645 krb5kdc[32250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.115.1@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:39:11.011058 791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:11.001841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:43103 (local address 127.23.115.1:60159)
0620 18:39:11.002411 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:11.002439 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:11.002568 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:11.002887 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:11.002891 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:11.003498 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:11.003688 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:11.003698 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:11.004763 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:11.004769 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:11.005693 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:11.005706 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:11.006393 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:11.006398 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:11.006571 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:11.007207 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:11.007225 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:11.010750 (+ 3525us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.115.62:43103: Server oryx/127.23.115.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":302,"threads_started":1}
W20260620 18:39:11.011510 792 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:11.001926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.1:60159 (local address 127.23.115.62:43103)
0620 18:39:11.002574 (+ 648us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:11.002580 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:11.002598 (+ 18us) server_negotiation.cc:407] Connection header received
0620 18:39:11.002679 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:11.002685 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:11.002750 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:11.002834 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:11.003852 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:11.004613 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:11.005833 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:11.006244 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:11.006349 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:11.011067 (+ 4718us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.115.1:60159: BlockingRecv error: recv got EOF from 127.23.115.1:60159 (error 108)
Metrics: {"server-negotiator.queue_time_us":504,"thread_start_us":369,"threads_started":1}
Jun 20 18:39:11 dist-test-slave-d645 krb5kdc[32250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.115.3@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:39:11 dist-test-slave-d645 krb5kdc[32250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.115.3@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:39:11.085759 793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:11.077248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:43103 (local address 127.23.115.3:35273)
0620 18:39:11.077815 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:11.077831 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:11.077957 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:11.078281 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:11.078286 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:11.078713 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:11.078854 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:11.078863 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:11.079874 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:11.079880 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:11.080799 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:11.080810 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:11.081510 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:11.081516 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:11.081663 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:11.082222 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:11.082239 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:11.085525 (+ 3286us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.115.62:43103: Server oryx/127.23.115.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":332,"threads_started":1}
W20260620 18:39:11.086014 792 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:11.077403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.3:35273 (local address 127.23.115.62:43103)
0620 18:39:11.077567 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:11.077573 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:11.077931 (+ 358us) server_negotiation.cc:407] Connection header received
0620 18:39:11.078072 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:11.078077 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:11.078156 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:11.078234 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:11.078998 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:11.079741 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:11.080942 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:11.081340 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:11.081441 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:11.085758 (+ 4317us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.115.3:35273: BlockingRecv error: recv got EOF from 127.23.115.3:35273 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Jun 20 18:39:11 dist-test-slave-d645 krb5kdc[32250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.115.2@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:39:11 dist-test-slave-d645 krb5kdc[32250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.115.2@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:39:11.398578 794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:11.390169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:43103 (local address 127.23.115.2:53717)
0620 18:39:11.390663 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:11.390680 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:11.390804 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:11.391141 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:11.391145 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:11.391541 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:11.391726 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:11.391734 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:11.392725 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:11.392729 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:11.393667 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:11.393678 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:11.394334 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:11.394340 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:11.394496 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:11.395085 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:11.395101 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:11.398202 (+ 3101us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.115.62:43103: Server oryx/127.23.115.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":262,"threads_started":1}
W20260620 18:39:11.398643 792 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:11.390276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.2:53717 (local address 127.23.115.62:43103)
0620 18:39:11.390479 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:11.390485 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:11.390754 (+ 269us) server_negotiation.cc:407] Connection header received
0620 18:39:11.390917 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:11.390922 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:11.391021 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:11.391100 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:11.391859 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:11.392593 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:11.393793 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:11.394199 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:11.394299 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:11.398412 (+ 4113us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.115.2:53717: BlockingRecv error: recv got EOF from 127.23.115.2:53717 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Jun 20 18:39:12 dist-test-slave-d645 krb5kdc[32250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.115.1@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:39:12 dist-test-slave-d645 krb5kdc[32250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.115.1@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:39:12.021346 795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:12.011990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:43103 (local address 127.23.115.1:34781)
0620 18:39:12.012587 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:12.012610 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:12.012720 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:12.013098 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:12.013102 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:12.013451 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:12.013643 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:12.013653 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:12.014854 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:12.014858 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:12.015867 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:12.015883 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:12.016639 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:12.016644 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:12.016817 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:12.017493 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:12.017517 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:12.021118 (+ 3601us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.115.62:43103: Server oryx/127.23.115.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":283,"threads_started":1}
W20260620 18:39:12.021647 796 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:12.012241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.1:34781 (local address 127.23.115.62:43103)
0620 18:39:12.012805 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:12.012810 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:12.012825 (+ 15us) server_negotiation.cc:407] Connection header received
0620 18:39:12.012872 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:12.012875 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:12.012940 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:12.013061 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:12.013814 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:12.014707 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:12.016017 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:12.016476 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:12.016641 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:12.021360 (+ 4719us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.115.1:34781: BlockingRecv error: recv got EOF from 127.23.115.1:34781 (error 108)
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":353,"threads_started":1}
Jun 20 18:39:12 dist-test-slave-d645 krb5kdc[32250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.115.3@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:39:12 dist-test-slave-d645 krb5kdc[32250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.115.3@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:39:12.096066 797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:12.086749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:43103 (local address 127.23.115.3:35099)
0620 18:39:12.087249 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:12.087266 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:12.087382 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:12.087673 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:12.087677 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:12.088040 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:12.088219 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:12.088227 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:12.089238 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:12.089242 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:12.090245 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:12.090260 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:12.091146 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:12.091153 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:12.091317 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:12.092027 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:12.092047 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:12.095817 (+ 3770us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.115.62:43103: Server oryx/127.23.115.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":282,"threads_started":1}
W20260620 18:39:12.096190 796 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:12.086851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.3:35099 (local address 127.23.115.62:43103)
0620 18:39:12.087097 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:12.087103 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:12.087342 (+ 239us) server_negotiation.cc:407] Connection header received
0620 18:39:12.087497 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:12.087500 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:12.087562 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:12.087640 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:12.088372 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:12.089123 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:12.090393 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:12.090986 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:12.091085 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:12.096006 (+ 4921us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.115.3:35099: BlockingRecv error: recv got EOF from 127.23.115.3:35099 (error 108)
Metrics: {"server-negotiator.queue_time_us":107}
Jun 20 18:39:12 dist-test-slave-d645 krb5kdc[32250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.115.2@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:39:12 dist-test-slave-d645 krb5kdc[32250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.115.2@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:39:12.408145 798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:12.399553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:43103 (local address 127.23.115.2:56493)
0620 18:39:12.400073 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:12.400089 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:12.400200 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:12.400524 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:12.400527 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:12.400866 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:12.401028 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:12.401036 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:12.402123 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:12.402127 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:12.403113 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:12.403127 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:12.403871 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:12.403876 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:12.403997 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:12.404611 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:12.404629 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:12.407815 (+ 3186us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.115.62:43103: Server oryx/127.23.115.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":255,"threads_started":1}
W20260620 18:39:12.408521 796 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:12.399659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.2:56493 (local address 127.23.115.62:43103)
0620 18:39:12.399824 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:12.399834 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:12.400167 (+ 333us) server_negotiation.cc:407] Connection header received
0620 18:39:12.400296 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:12.400300 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:12.400362 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:12.400469 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:12.401180 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:12.401990 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:12.403250 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:12.403733 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:12.403842 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:12.408168 (+ 4326us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.115.2:56493: BlockingRecv error: recv got EOF from 127.23.115.2:56493 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 20 18:39:13 dist-test-slave-d645 krb5kdc[32250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.115.1@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:39:13 dist-test-slave-d645 krb5kdc[32250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.115.1@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:39:13.032708 799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:13.022414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:43103 (local address 127.23.115.1:52943)
0620 18:39:13.022986 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:13.023011 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:13.023150 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:13.023550 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:13.023555 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:13.024026 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:13.024224 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:13.024236 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:13.025258 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:13.025262 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:13.026269 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:13.026282 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:13.027010 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:13.027015 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:13.027165 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:13.027899 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:13.027946 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:13.032354 (+ 4408us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.115.62:43103: Server oryx/127.23.115.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":269,"threads_started":1}
W20260620 18:39:13.032879 800 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:13.022528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.1:52943 (local address 127.23.115.62:43103)
0620 18:39:13.023249 (+ 721us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:13.023254 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:13.023270 (+ 16us) server_negotiation.cc:407] Connection header received
0620 18:39:13.023321 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:13.023324 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:13.023379 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:13.023467 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:13.024390 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:13.025127 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:13.026417 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:13.026817 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:13.026914 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:13.032649 (+ 5735us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.115.1:52943: BlockingRecv error: recv got EOF from 127.23.115.1:52943 (error 108)
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":485,"threads_started":1}
Jun 20 18:39:13 dist-test-slave-d645 krb5kdc[32250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.115.3@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:39:13 dist-test-slave-d645 krb5kdc[32250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.115.3@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:39:13.107139 801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:13.097078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:43103 (local address 127.23.115.3:44665)
0620 18:39:13.097662 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:13.097679 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:13.097797 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:13.098121 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:13.098125 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:13.098483 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:13.098685 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:13.098694 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:13.099759 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:13.099765 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:13.100778 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:13.100790 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:13.101565 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:13.101570 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:13.101711 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:13.102274 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:13.102293 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:13.106796 (+ 4503us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.115.62:43103: Server oryx/127.23.115.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":324,"threads_started":1}
W20260620 18:39:13.107282 800 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:13.097218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.3:44665 (local address 127.23.115.62:43103)
0620 18:39:13.097434 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:13.097439 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:13.097768 (+ 329us) server_negotiation.cc:407] Connection header received
0620 18:39:13.097895 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:13.097899 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:13.097963 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:13.098053 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:13.098831 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:13.099612 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:13.100937 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:13.101412 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:13.101504 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:13.107035 (+ 5531us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.115.3:44665: BlockingRecv error: recv got EOF from 127.23.115.3:44665 (error 108)
Metrics: {"server-negotiator.queue_time_us":112}
Jun 20 18:39:13 dist-test-slave-d645 krb5kdc[32250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.115.2@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:39:13 dist-test-slave-d645 krb5kdc[32250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.115.2@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:39:13.418061 802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:13.409427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:43103 (local address 127.23.115.2:49651)
0620 18:39:13.409903 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:13.409918 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:13.410054 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:13.410379 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:13.410384 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:13.410774 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:13.410923 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:13.410960 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:13.411962 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:13.411966 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:13.412929 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:13.412941 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:13.413621 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:13.413627 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:13.413782 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:13.414309 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:13.414325 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:13.417776 (+ 3451us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.115.62:43103: Server oryx/127.23.115.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":283,"threads_started":1}
W20260620 18:39:13.418136 800 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:13.409488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.2:49651 (local address 127.23.115.62:43103)
0620 18:39:13.409676 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:13.409682 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:13.410000 (+ 318us) server_negotiation.cc:407] Connection header received
0620 18:39:13.410177 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:13.410181 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:13.410254 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:13.410332 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:13.411086 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:13.411823 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:13.413065 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:13.413479 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:13.413583 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:13.417910 (+ 4327us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.115.2:49651: BlockingRecv error: recv got EOF from 127.23.115.2:49651 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 20 18:39:14 dist-test-slave-d645 krb5kdc[32250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.115.1@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:39:14 dist-test-slave-d645 krb5kdc[32250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.115.1@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:39:14.042872 803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:14.033705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:43103 (local address 127.23.115.1:39883)
0620 18:39:14.034227 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:14.034249 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:14.034355 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:14.034713 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:14.034716 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:14.035198 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:14.035433 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:14.035452 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:14.036598 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:14.036602 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:14.037522 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:14.037534 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:14.038244 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:14.038249 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:14.038380 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:14.039033 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:14.039062 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:14.042471 (+ 3409us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.115.62:43103: Server oryx/127.23.115.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":241,"threads_started":1}
W20260620 18:39:14.042999 804 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:14.033811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.1:39883 (local address 127.23.115.62:43103)
0620 18:39:14.034306 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:14.034312 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:14.034333 (+ 21us) server_negotiation.cc:407] Connection header received
0620 18:39:14.034480 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:14.034484 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:14.034557 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:14.034663 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:14.035635 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:14.036468 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:14.037695 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:14.038087 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:14.038185 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:14.042734 (+ 4549us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.115.1:39883: BlockingRecv error: recv got EOF from 127.23.115.1:39883 (error 108)
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":258,"threads_started":1}
Jun 20 18:39:14 dist-test-slave-d645 krb5kdc[32250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.115.3@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:39:14 dist-test-slave-d645 krb5kdc[32250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.115.3@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:39:14.117551 805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:14.108244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:43103 (local address 127.23.115.3:55833)
0620 18:39:14.108897 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:14.108912 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:14.109041 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:14.109610 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:14.109614 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:14.110098 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:14.110277 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:14.110286 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:14.111407 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:14.111411 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:14.112413 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:14.112425 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:14.113120 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:14.113125 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:14.113250 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:14.113875 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:14.113893 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:14.117277 (+ 3384us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.115.62:43103: Server oryx/127.23.115.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":429,"threads_started":1}
W20260620 18:39:14.117707 804 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:14.108368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.3:55833 (local address 127.23.115.62:43103)
0620 18:39:14.108585 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:14.108591 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:14.108995 (+ 404us) server_negotiation.cc:407] Connection header received
0620 18:39:14.109312 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:14.109317 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:14.109394 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:14.109497 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:14.110433 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:14.111216 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:14.112585 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:14.112985 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:14.113080 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:14.117502 (+ 4422us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.115.3:55833: BlockingRecv error: recv got EOF from 127.23.115.3:55833 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 20 18:39:14 dist-test-slave-d645 krb5kdc[32250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.115.2@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:39:14 dist-test-slave-d645 krb5kdc[32250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.115.2@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:39:14.428107 806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:14.419214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:43103 (local address 127.23.115.2:48821)
0620 18:39:14.419671 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:14.419687 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:14.419798 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:14.420157 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:14.420165 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:14.420535 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:14.420681 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:14.420689 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:14.421753 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:14.421758 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:14.422691 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:14.422703 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:14.423431 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:14.423435 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:14.423561 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:14.424135 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:14.424152 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:14.427877 (+ 3725us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.115.62:43103: Server oryx/127.23.115.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":239,"threads_started":1}
W20260620 18:39:14.428253 804 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:14.419430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.2:48821 (local address 127.23.115.62:43103)
0620 18:39:14.419586 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:14.419590 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:14.419777 (+ 187us) server_negotiation.cc:407] Connection header received
0620 18:39:14.419931 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:14.419935 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:14.420009 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:14.420099 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:14.420837 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:14.421604 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:14.422828 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:14.423291 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:14.423385 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:14.428045 (+ 4660us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.115.2:48821: BlockingRecv error: recv got EOF from 127.23.115.2:48821 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 20 18:39:15 dist-test-slave-d645 krb5kdc[32250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.115.1@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:39:15 dist-test-slave-d645 krb5kdc[32250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.115.1@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:39:15.056164 807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:15.044075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:43103 (local address 127.23.115.1:51541)
0620 18:39:15.044567 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:15.044591 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:15.044715 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:15.045122 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:15.045128 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:15.045562 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:15.045725 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:15.045734 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:15.046852 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:15.046857 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:15.047845 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:15.047860 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:15.048684 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:15.048689 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:15.048819 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:15.052408 (+ 3589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:15.052426 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:15.055891 (+ 3465us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.115.62:43103: Server oryx/127.23.115.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":258,"threads_started":1}
W20260620 18:39:15.056391 808 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:15.044164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.1:51541 (local address 127.23.115.62:43103)
0620 18:39:15.044598 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:15.044603 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:15.044664 (+ 61us) server_negotiation.cc:407] Connection header received
0620 18:39:15.044856 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:15.044861 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:15.044950 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:15.045041 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:15.045880 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:15.046707 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:15.048002 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:15.048415 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:15.048562 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:15.056139 (+ 7577us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.115.1:51541: BlockingRecv error: recv got EOF from 127.23.115.1:51541 (error 108)
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":236,"threads_started":1}
Jun 20 18:39:15 dist-test-slave-d645 krb5kdc[32250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.115.3@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:39:15 dist-test-slave-d645 krb5kdc[32250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.115.3@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:39:15.128377 809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:15.118429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:43103 (local address 127.23.115.3:37693)
0620 18:39:15.118869 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:15.118885 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:15.119036 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:15.119389 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:15.119392 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:15.119770 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:15.119911 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:15.119920 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:15.121010 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:15.121014 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:15.122014 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:15.122049 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:15.122782 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:15.122786 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:15.122922 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:15.123543 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:15.123565 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:15.128107 (+ 4542us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.115.62:43103: Server oryx/127.23.115.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":227,"threads_started":1}
W20260620 18:39:15.128929 808 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:15.118565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.3:37693 (local address 127.23.115.62:43103)
0620 18:39:15.118812 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:15.118818 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:15.118984 (+ 166us) server_negotiation.cc:407] Connection header received
0620 18:39:15.119175 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:15.119179 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:15.119253 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:15.119336 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:15.120087 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:15.120866 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:15.122183 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:15.122635 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:15.122731 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:15.128537 (+ 5806us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.115.3:37693: BlockingRecv error: recv got EOF from 127.23.115.3:37693 (error 108)
Metrics: {"server-negotiator.queue_time_us":108}
Jun 20 18:39:15 dist-test-slave-d645 krb5kdc[32250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.115.2@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:39:15 dist-test-slave-d645 krb5kdc[32250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.115.2@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:39:15.438025 810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:15.429017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:43103 (local address 127.23.115.2:38501)
0620 18:39:15.429493 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:15.429507 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:15.429673 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:15.430010 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:15.430015 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:15.430417 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:15.430583 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:15.430591 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:15.431709 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:15.431713 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:15.432640 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:15.432651 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:15.433318 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:15.433324 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:15.433483 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:15.434039 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:15.434055 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:15.437687 (+ 3632us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.115.62:43103: Server oryx/127.23.115.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":250,"threads_started":1}
W20260620 18:39:15.438210 808 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:15.429124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.2:38501 (local address 127.23.115.62:43103)
0620 18:39:15.429314 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:15.429320 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:15.429620 (+ 300us) server_negotiation.cc:407] Connection header received
0620 18:39:15.429789 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:15.429794 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:15.429874 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:15.429971 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:15.430718 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:15.431579 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:15.432773 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:15.433176 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:15.433282 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:15.437940 (+ 4658us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.115.2:38501: BlockingRecv error: recv got EOF from 127.23.115.2:38501 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 20 18:39:16 dist-test-slave-d645 krb5kdc[32250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.115.1@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:39:16 dist-test-slave-d645 krb5kdc[32250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.115.1@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:39:16.066428 812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:16.057078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:43103 (local address 127.23.115.1:55857)
0620 18:39:16.057662 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:16.057688 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:16.057832 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:16.058204 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:16.058207 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:16.058626 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:16.058797 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:16.058806 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:16.059902 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:16.059906 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:16.060931 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:16.060944 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:16.061845 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:16.061854 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:16.062038 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:16.062809 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:16.062830 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:16.066164 (+ 3334us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.115.62:43103: Server oryx/127.23.115.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":270,"threads_started":1}
W20260620 18:39:16.066776 811 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:16.057143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.1:55857 (local address 127.23.115.62:43103)
0620 18:39:16.057635 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:16.057641 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:16.057768 (+ 127us) server_negotiation.cc:407] Connection header received
0620 18:39:16.057951 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:16.057956 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:16.058033 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:16.058131 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:16.058982 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:16.059761 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:16.061083 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:16.061542 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:16.061648 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:16.066447 (+ 4799us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.115.1:55857: BlockingRecv error: recv got EOF from 127.23.115.1:55857 (error 108)
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":245,"threads_started":1}
Jun 20 18:39:16 dist-test-slave-d645 krb5kdc[32250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.115.3@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:39:16 dist-test-slave-d645 krb5kdc[32250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.115.3@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:39:16.138809 813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:16.129498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:43103 (local address 127.23.115.3:38241)
0620 18:39:16.130012 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:16.130027 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:16.130159 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:16.130521 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:16.130526 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:16.130986 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:16.131156 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:16.131166 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:16.132297 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:16.132301 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:16.133349 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:16.133361 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:16.134010 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:16.134015 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:16.134170 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:16.134796 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:16.134815 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:16.138480 (+ 3665us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.115.62:43103: Server oryx/127.23.115.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":269,"threads_started":1}
W20260620 18:39:16.139091 811 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:16.129659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.3:38241 (local address 127.23.115.62:43103)
0620 18:39:16.129888 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:16.129894 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:16.130105 (+ 211us) server_negotiation.cc:407] Connection header received
0620 18:39:16.130278 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:16.130283 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:16.130350 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:16.130473 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:16.131316 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:16.132169 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:16.133484 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:16.133867 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:16.133972 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:16.138796 (+ 4824us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.115.3:38241: BlockingRecv error: recv got EOF from 127.23.115.3:38241 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Jun 20 18:39:16 dist-test-slave-d645 krb5kdc[32250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.115.2@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:39:16 dist-test-slave-d645 krb5kdc[32250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.115.2@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:39:16.448339 814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:16.439007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:43103 (local address 127.23.115.2:54745)
0620 18:39:16.439545 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:16.439561 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:16.439686 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:16.439976 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:16.439980 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:16.440381 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:16.440548 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:16.440557 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:16.441628 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:16.441632 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:16.442567 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:16.442578 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:16.443558 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:16.443562 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:16.443685 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:16.444280 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:16.444306 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:16.448106 (+ 3800us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.115.62:43103: Server oryx/127.23.115.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":228,"threads_started":1}
W20260620 18:39:16.448669 811 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:16.439041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.2:54745 (local address 127.23.115.62:43103)
0620 18:39:16.439226 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:16.439232 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:16.439667 (+ 435us) server_negotiation.cc:407] Connection header received
0620 18:39:16.439786 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:16.439790 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:16.439867 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:16.439969 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:16.440722 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:16.441488 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:16.442796 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:16.443391 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:16.443495 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:16.448299 (+ 4804us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.115.2:54745: BlockingRecv error: recv got EOF from 127.23.115.2:54745 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 20 18:39:17 dist-test-slave-d645 krb5kdc[32250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.115.1@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:39:17 dist-test-slave-d645 krb5kdc[32250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.115.1@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:39:17.077160 815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:17.067495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:43103 (local address 127.23.115.1:60015)
0620 18:39:17.068075 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:17.068096 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:17.068207 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:17.068637 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:17.068641 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:17.069136 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:17.069316 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:17.069324 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:17.070597 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:17.070602 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:17.071821 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:17.071834 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:17.072725 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:17.072730 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:17.072843 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:17.073525 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:17.073543 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:17.076904 (+ 3361us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.115.62:43103: Server oryx/127.23.115.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":268,"threads_started":1}
W20260620 18:39:17.077411 816 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:17.067647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.1:60015 (local address 127.23.115.62:43103)
0620 18:39:17.068120 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:17.068127 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:17.068149 (+ 22us) server_negotiation.cc:407] Connection header received
0620 18:39:17.068351 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:17.068355 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:17.068485 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:17.068580 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:17.069496 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:17.070411 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:17.071988 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:17.072545 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:17.072652 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:17.077168 (+ 4516us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.115.1:60015: BlockingRecv error: recv got EOF from 127.23.115.1:60015 (error 108)
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":245,"threads_started":1}
Jun 20 18:39:17 dist-test-slave-d645 krb5kdc[32250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.115.3@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:39:17 dist-test-slave-d645 krb5kdc[32250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.115.3@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:39:17.148859 817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:17.139841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:43103 (local address 127.23.115.3:35551)
0620 18:39:17.140449 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:17.140465 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:17.140659 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:17.140962 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:17.140966 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:17.141423 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:17.141598 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:17.141607 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:17.142606 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:17.142610 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:17.143792 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:17.143808 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:17.144659 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:17.144665 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:17.144808 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:17.145382 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:17.145399 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:17.148641 (+ 3242us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.115.62:43103: Server oryx/127.23.115.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":297,"threads_started":1}
W20260620 18:39:17.149024 816 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:17.140024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.3:35551 (local address 127.23.115.62:43103)
0620 18:39:17.140238 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:17.140244 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:17.140592 (+ 348us) server_negotiation.cc:407] Connection header received
0620 18:39:17.140778 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:17.140781 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:17.140842 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:17.140921 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:17.141734 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:17.142472 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:17.143942 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:17.144429 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:17.144528 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:17.148832 (+ 4304us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.115.3:35551: BlockingRecv error: recv got EOF from 127.23.115.3:35551 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 20 18:39:17 dist-test-slave-d645 krb5kdc[32250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.115.2@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:39:17 dist-test-slave-d645 krb5kdc[32250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.115.2@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:39:17.458745 818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:17.449285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:43103 (local address 127.23.115.2:58447)
0620 18:39:17.449824 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:17.449841 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:17.449955 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:17.450331 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:17.450335 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:17.450788 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:17.450998 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:17.451009 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:17.452228 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:17.452233 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:17.453248 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:17.453261 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:17.454043 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:17.454049 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:17.454167 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:17.454818 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:17.454842 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:17.458490 (+ 3648us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.115.62:43103: Server oryx/127.23.115.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":292,"threads_started":1}
W20260620 18:39:17.459103 816 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:17.449486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.2:58447 (local address 127.23.115.62:43103)
0620 18:39:17.449647 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:17.449651 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:17.449931 (+ 280us) server_negotiation.cc:407] Connection header received
0620 18:39:17.450070 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:17.450074 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:17.450151 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:17.450240 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:17.451150 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:17.452089 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:17.453396 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:17.453855 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:17.453957 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:17.458668 (+ 4711us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.115.2:58447: BlockingRecv error: recv got EOF from 127.23.115.2:58447 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 20 18:39:18 dist-test-slave-d645 krb5kdc[32250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.115.1@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:39:18 dist-test-slave-d645 krb5kdc[32250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.115.1@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:39:18.087777 819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:18.078170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:43103 (local address 127.23.115.1:39049)
0620 18:39:18.078678 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:18.078699 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:18.078797 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:18.079123 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:18.079127 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:18.079484 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:18.079662 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:18.079671 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:18.080767 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:18.080772 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:18.081826 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:18.081843 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:18.082808 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:18.082816 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:18.083056 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:18.083686 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:18.083705 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:18.087490 (+ 3785us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.115.62:43103: Server oryx/127.23.115.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":275,"threads_started":1}
W20260620 18:39:18.088032 820 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:18.078346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.1:39049 (local address 127.23.115.62:43103)
0620 18:39:18.078813 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:18.078817 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:18.078835 (+ 18us) server_negotiation.cc:407] Connection header received
0620 18:39:18.078893 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:18.078897 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:18.078980 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:18.079072 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:18.079843 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:18.080611 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:18.081952 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:18.082607 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:18.082754 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:18.087732 (+ 4978us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.115.1:39049: BlockingRecv error: recv got EOF from 127.23.115.1:39049 (error 108)
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":235,"threads_started":1}
Jun 20 18:39:18 dist-test-slave-d645 krb5kdc[32250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.115.3@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:39:18 dist-test-slave-d645 krb5kdc[32250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.115.3@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:39:18.158172 821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:18.149743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:43103 (local address 127.23.115.3:38795)
0620 18:39:18.150265 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:18.150281 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:18.150390 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:18.150704 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:18.150707 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:18.151134 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:18.151290 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:18.151299 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:18.152375 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:18.152380 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:18.153312 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:18.153325 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:18.153971 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:18.153975 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:18.154088 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:18.154663 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:18.154681 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:18.157929 (+ 3248us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.115.62:43103: Server oryx/127.23.115.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":272,"threads_started":1}
W20260620 18:39:18.158408 820 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:18.149835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.3:38795 (local address 127.23.115.62:43103)
0620 18:39:18.149987 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:18.149992 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:18.150345 (+ 353us) server_negotiation.cc:407] Connection header received
0620 18:39:18.150516 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:18.150520 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:18.150581 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:18.150668 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:18.151422 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:18.152224 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:18.153432 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:18.153824 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:18.153910 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:18.158185 (+ 4275us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.115.3:38795: BlockingRecv error: recv got EOF from 127.23.115.3:38795 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 20 18:39:18 dist-test-slave-d645 krb5kdc[32250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.115.2@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:39:18 dist-test-slave-d645 krb5kdc[32250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.115.2@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:39:18.469132 822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:18.459921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:43103 (local address 127.23.115.2:53317)
0620 18:39:18.460379 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:18.460395 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:18.460534 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:18.460908 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:18.460913 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:18.461318 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:18.461459 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:18.461467 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:18.462622 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:18.462627 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:18.463728 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:18.463743 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:18.464410 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:18.464416 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:18.464594 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:18.465256 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:18.465274 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:18.468904 (+ 3630us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.115.62:43103: Server oryx/127.23.115.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":229,"threads_started":1}
W20260620 18:39:18.469357 820 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:18.459934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.2:53317 (local address 127.23.115.62:43103)
0620 18:39:18.460122 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:18.460129 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:18.460487 (+ 358us) server_negotiation.cc:407] Connection header received
0620 18:39:18.460706 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:18.460710 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:18.460784 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:18.460863 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:18.461611 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:18.462466 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:18.463860 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:18.464272 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:18.464375 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:18.469117 (+ 4742us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.115.2:53317: BlockingRecv error: recv got EOF from 127.23.115.2:53317 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 20 18:39:19 dist-test-slave-d645 krb5kdc[32250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.115.1@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:39:19 dist-test-slave-d645 krb5kdc[32250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.115.1@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:39:19.097647 823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:19.088753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:43103 (local address 127.23.115.1:34603)
0620 18:39:19.089281 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:19.089306 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:19.089444 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:19.089830 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:19.089835 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:19.090275 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:19.090436 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:19.090446 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:19.091508 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:19.091513 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:19.092473 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:19.092484 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:19.093202 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:19.093208 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:19.093363 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:19.093944 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:19.093965 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:19.097389 (+ 3424us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.115.62:43103: Server oryx/127.23.115.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":263,"threads_started":1}
W20260620 18:39:19.097872 824 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:19.088815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.1:34603 (local address 127.23.115.62:43103)
0620 18:39:19.089297 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:19.089301 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:19.089368 (+ 67us) server_negotiation.cc:407] Connection header received
0620 18:39:19.089601 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:19.089607 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:19.089696 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:19.089780 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:19.090586 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:19.091371 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:19.092612 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:19.093063 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:19.093167 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:19.097636 (+ 4469us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.115.1:34603: BlockingRecv error: recv got EOF from 127.23.115.1:34603 (error 108)
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":235,"threads_started":1}
Jun 20 18:39:19 dist-test-slave-d645 krb5kdc[32250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.115.3@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:39:19 dist-test-slave-d645 krb5kdc[32250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.115.3@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:39:19.168468 825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:19.159231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:43103 (local address 127.23.115.3:52295)
0620 18:39:19.159788 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:19.159804 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:19.159934 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:19.160264 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:19.160269 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:19.160727 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:19.160866 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:19.160874 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:19.161924 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:19.161930 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:19.163155 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:19.163168 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:19.163950 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:19.163955 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:19.164091 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:19.164720 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:19.164738 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:19.168159 (+ 3421us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.115.62:43103: Server oryx/127.23.115.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":278,"threads_started":1}
W20260620 18:39:19.168587 824 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:19.159323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.3:52295 (local address 127.23.115.62:43103)
0620 18:39:19.159504 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:19.159510 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:19.159899 (+ 389us) server_negotiation.cc:407] Connection header received
0620 18:39:19.160057 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:19.160062 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:19.160125 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:19.160203 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:19.161031 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:19.161760 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:19.163299 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:19.163802 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:19.163900 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:19.168357 (+ 4457us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.115.3:52295: BlockingRecv error: recv got EOF from 127.23.115.3:52295 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 20 18:39:19 dist-test-slave-d645 krb5kdc[32250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.115.2@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:39:19 dist-test-slave-d645 krb5kdc[32250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.115.2@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:39:19.479417 826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:19.470116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:43103 (local address 127.23.115.2:37773)
0620 18:39:19.470681 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:19.470703 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:19.470841 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:19.471197 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:19.471202 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:19.471585 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:19.471761 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:19.471770 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:19.472828 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:19.472833 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:19.473849 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:19.473861 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:19.474596 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:19.474602 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:19.474766 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:19.475360 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:19.475378 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:19.479068 (+ 3690us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.115.62:43103: Server oryx/127.23.115.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":264,"threads_started":1}
W20260620 18:39:19.479833 824 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:19.470165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.2:37773 (local address 127.23.115.62:43103)
0620 18:39:19.470355 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:19.470361 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:19.470796 (+ 435us) server_negotiation.cc:407] Connection header received
0620 18:39:19.470984 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:19.470990 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:19.471069 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:19.471157 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:19.471906 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:19.472673 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:19.473992 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:19.474421 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:19.474527 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:19.479340 (+ 4813us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.115.2:37773: BlockingRecv error: recv got EOF from 127.23.115.2:37773 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 20 18:39:20 dist-test-slave-d645 krb5kdc[32250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.115.1@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:39:20 dist-test-slave-d645 krb5kdc[32250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.115.1@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:39:20.112067 827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:20.098622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:43103 (local address 127.23.115.1:58191)
0620 18:39:20.099173 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:20.099201 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:20.099307 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:20.099611 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:20.099616 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:20.100019 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:20.100226 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:20.100234 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:20.101356 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:20.101360 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:20.102434 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:20.102447 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:20.103313 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:20.103318 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:20.103435 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:20.108156 (+ 4721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:20.108176 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:20.111850 (+ 3674us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.115.62:43103: Server oryx/127.23.115.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":268,"threads_started":1}
W20260620 18:39:20.112352 828 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:20.098813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.1:58191 (local address 127.23.115.62:43103)
0620 18:39:20.099304 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:20.099308 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:20.099324 (+ 16us) server_negotiation.cc:407] Connection header received
0620 18:39:20.099388 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:20.099391 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:20.099448 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:20.099534 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:20.100415 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:20.101220 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:20.102588 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:20.103172 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:20.103264 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:20.112100 (+ 8836us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.115.1:58191: BlockingRecv error: recv got EOF from 127.23.115.1:58191 (error 108)
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":264,"threads_started":1}
Jun 20 18:39:20 dist-test-slave-d645 krb5kdc[32250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.115.3@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:39:20 dist-test-slave-d645 krb5kdc[32250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.115.3@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:39:20.180238 829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:20.169511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:43103 (local address 127.23.115.3:46405)
0620 18:39:20.170069 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:20.170087 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:20.170223 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:20.170601 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:20.170605 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:20.171045 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:20.171214 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:20.171224 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:20.172847 (+ 1623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:20.172853 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:20.174110 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:20.174124 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:20.175043 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:20.175051 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:20.175224 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:20.175989 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:20.176012 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:20.179975 (+ 3963us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.115.62:43103: Server oryx/127.23.115.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":286,"threads_started":1}
W20260620 18:39:20.180395 828 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:20.169624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.3:46405 (local address 127.23.115.62:43103)
0620 18:39:20.169785 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:20.169790 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:20.170210 (+ 420us) server_negotiation.cc:407] Connection header received
0620 18:39:20.170357 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:20.170361 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:20.170424 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:20.170527 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:20.171386 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:20.172689 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:20.174260 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:20.174703 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:20.174811 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:20.180217 (+ 5406us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.115.3:46405: BlockingRecv error: recv got EOF from 127.23.115.3:46405 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 20 18:39:20 dist-test-slave-d645 krb5kdc[32250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.115.2@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:39:20 dist-test-slave-d645 krb5kdc[32250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.115.2@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:39:20.489753 830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:20.480482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:43103 (local address 127.23.115.2:52301)
0620 18:39:20.480982 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:20.481004 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:20.481171 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:20.481605 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:20.481611 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:20.482139 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:20.482395 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:20.482410 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:20.483506 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:20.483510 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:20.484454 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:20.484465 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:20.485333 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:20.485338 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:20.485510 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:20.486145 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:20.486163 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:20.489528 (+ 3365us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.115.62:43103: Server oryx/127.23.115.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":255,"threads_started":1}
W20260620 18:39:20.490092 828 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:20.480646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.2:52301 (local address 127.23.115.62:43103)
0620 18:39:20.480847 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:20.480853 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:20.481106 (+ 253us) server_negotiation.cc:407] Connection header received
0620 18:39:20.481316 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:20.481323 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:20.481423 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:20.481531 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:20.482553 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:20.483368 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:20.484620 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:20.485180 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:20.485299 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:20.489753 (+ 4454us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.115.2:52301: BlockingRecv error: recv got EOF from 127.23.115.2:52301 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Jun 20 18:39:21 dist-test-slave-d645 krb5kdc[32250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.115.1@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:39:21 dist-test-slave-d645 krb5kdc[32250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.115.1@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:39:21.123842 831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:21.113347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:43103 (local address 127.23.115.1:59973)
0620 18:39:21.114132 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:21.114156 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:21.114305 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:21.114654 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:21.114658 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:21.115119 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:21.115282 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:21.115292 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:21.116368 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:21.116373 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:21.117395 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:21.117408 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:21.118261 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:21.118267 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:21.118433 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:21.119172 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:21.119339 (+ 167us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:21.123507 (+ 4168us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.115.62:43103: Server oryx/127.23.115.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":665,"thread_start_us":465,"threads_started":1}
W20260620 18:39:21.123981 832 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:21.113508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.1:59973 (local address 127.23.115.62:43103)
0620 18:39:21.114019 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:21.114024 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:21.114253 (+ 229us) server_negotiation.cc:407] Connection header received
0620 18:39:21.114417 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:21.114422 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:21.114504 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:21.114588 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:21.115419 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:21.116244 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:21.117546 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:21.118087 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:21.118196 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:21.123755 (+ 5559us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.115.1:59973: BlockingRecv error: recv got EOF from 127.23.115.1:59973 (error 108)
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":271,"threads_started":1}
Jun 20 18:39:21 dist-test-slave-d645 krb5kdc[32250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.115.3@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:39:21 dist-test-slave-d645 krb5kdc[32250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.115.3@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:39:21.189698 833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:21.181123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:43103 (local address 127.23.115.3:52441)
0620 18:39:21.181571 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:21.181588 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:21.181708 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:21.181969 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:21.181972 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:21.182401 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:21.182582 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:21.182591 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:21.183630 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:21.183633 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:21.184522 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:21.184533 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:21.185139 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:21.185144 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:21.185266 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:21.185859 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:21.185876 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:21.189437 (+ 3561us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.115.62:43103: Server oryx/127.23.115.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":247,"threads_started":1}
W20260620 18:39:21.189795 832 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:21.181223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.3:52441 (local address 127.23.115.62:43103)
0620 18:39:21.181377 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:21.181381 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:21.181662 (+ 281us) server_negotiation.cc:407] Connection header received
0620 18:39:21.181798 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:21.181801 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:21.181863 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:21.181942 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:21.182687 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:21.183500 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:21.184627 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:21.184998 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:21.185085 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:21.189602 (+ 4517us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.115.3:52441: BlockingRecv error: recv got EOF from 127.23.115.3:52441 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jun 20 18:39:21 dist-test-slave-d645 krb5kdc[32250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.115.2@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:39:21 dist-test-slave-d645 krb5kdc[32250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.115.2@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:39:21.499639 834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:21.490756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:43103 (local address 127.23.115.2:39765)
0620 18:39:21.491299 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:21.491316 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:21.491463 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:21.491865 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:21.491870 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:21.492290 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:21.492454 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:21.492462 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:21.493503 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:21.493508 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:21.494416 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:21.494427 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:21.495133 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:21.495138 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:21.495312 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:21.495883 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:21.495900 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:21.499317 (+ 3417us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.115.62:43103: Server oryx/127.23.115.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":298,"threads_started":1}
W20260620 18:39:21.499788 832 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:21.490857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.2:39765 (local address 127.23.115.62:43103)
0620 18:39:21.491144 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:21.491151 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:21.491412 (+ 261us) server_negotiation.cc:407] Connection header received
0620 18:39:21.491628 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:21.491632 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:21.491724 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:21.491804 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:21.492626 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:21.493365 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:21.494554 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:21.494986 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:21.495107 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:21.499539 (+ 4432us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.115.2:39765: BlockingRecv error: recv got EOF from 127.23.115.2:39765 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
Jun 20 18:39:22 dist-test-slave-d645 krb5kdc[32250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.115.1@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:39:22 dist-test-slave-d645 krb5kdc[32250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.115.1@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:39:22.135890 835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:22.124867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:43103 (local address 127.23.115.1:48925)
0620 18:39:22.125603 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:22.125632 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:22.125746 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:22.126045 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:22.126049 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:22.126625 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:22.126800 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:22.126811 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:22.128592 (+ 1781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:22.128596 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:22.129692 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:22.129706 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:22.130557 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:22.130561 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:22.130687 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:22.131466 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:22.131484 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:22.135549 (+ 4065us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.115.62:43103: Server oryx/127.23.115.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":308,"threads_started":1}
W20260620 18:39:22.136026 836 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:22.125127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.1:48925 (local address 127.23.115.62:43103)
0620 18:39:22.125767 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:22.125771 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:22.125789 (+ 18us) server_negotiation.cc:407] Connection header received
0620 18:39:22.125833 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:22.125837 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:22.125906 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:22.125987 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:22.127297 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:22.128438 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:22.129884 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:22.130367 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:22.130471 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:22.135757 (+ 5286us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.115.1:48925: BlockingRecv error: recv got EOF from 127.23.115.1:48925 (error 108)
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":385,"threads_started":1}
Jun 20 18:39:22 dist-test-slave-d645 krb5kdc[32250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.115.3@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:39:22 dist-test-slave-d645 krb5kdc[32250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.115.3@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:39:22.200115 837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:22.190632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:43103 (local address 127.23.115.3:40379)
0620 18:39:22.191234 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:22.191250 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:22.191358 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:22.191619 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:22.191622 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:22.192022 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:22.192183 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:22.192191 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:22.193532 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:22.193539 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:22.194727 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:22.194750 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:22.195650 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:22.195655 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:22.195817 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:22.196434 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:22.196452 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:22.199864 (+ 3412us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.115.62:43103: Server oryx/127.23.115.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":356,"threads_started":1}
W20260620 18:39:22.200235 836 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:22.190753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.3:40379 (local address 127.23.115.62:43103)
0620 18:39:22.190970 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:22.190976 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:22.191331 (+ 355us) server_negotiation.cc:407] Connection header received
0620 18:39:22.191446 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:22.191450 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:22.191513 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:22.191595 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:22.192477 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:22.193268 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:22.194968 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:22.195419 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:22.195523 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:22.200045 (+ 4522us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.115.3:40379: BlockingRecv error: recv got EOF from 127.23.115.3:40379 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Jun 20 18:39:22 dist-test-slave-d645 krb5kdc[32250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.115.2@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:39:22 dist-test-slave-d645 krb5kdc[32250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.115.2@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:39:22.510226 838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:22.500793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:43103 (local address 127.23.115.2:36943)
0620 18:39:22.501317 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:22.501335 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:22.501460 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:22.501831 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:22.501835 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:22.502240 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:22.502392 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:22.502402 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:22.503468 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:22.503473 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:22.504408 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:22.504421 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:22.505264 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:22.505269 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:22.505453 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:22.506172 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:22.506191 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:22.509993 (+ 3802us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.115.62:43103: Server oryx/127.23.115.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":282,"threads_started":1}
W20260620 18:39:22.510481 836 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:22.501054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.2:36943 (local address 127.23.115.62:43103)
0620 18:39:22.501267 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:22.501273 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:22.501406 (+ 133us) server_negotiation.cc:407] Connection header received
0620 18:39:22.501580 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:22.501584 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:22.501662 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:22.501741 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:22.502510 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:22.503320 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:22.504551 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:22.505100 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:22.505246 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:22.510228 (+ 4982us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.115.2:36943: BlockingRecv error: recv got EOF from 127.23.115.2:36943 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Jun 20 18:39:23 dist-test-slave-d645 krb5kdc[32250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.115.1@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:39:23 dist-test-slave-d645 krb5kdc[32250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.115.1@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:39:23.146646 839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:23.137127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:43103 (local address 127.23.115.1:59759)
0620 18:39:23.137734 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:23.137759 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:23.138021 (+ 262us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:23.138384 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:23.138389 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:23.138761 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:23.139008 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:23.139022 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:23.140367 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:23.140371 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:23.141282 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:23.141293 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:23.142226 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:23.142230 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:23.142359 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:23.142964 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:23.142981 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:23.146365 (+ 3384us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.115.62:43103: Server oryx/127.23.115.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":252,"threads_started":1}
W20260620 18:39:23.146816 840 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:23.137279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.1:59759 (local address 127.23.115.62:43103)
0620 18:39:23.137988 (+ 709us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:23.137993 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:23.138014 (+ 21us) server_negotiation.cc:407] Connection header received
0620 18:39:23.138133 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:23.138138 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:23.138211 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:23.138333 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:23.139171 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:23.140216 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:23.141489 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:23.142052 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:23.142197 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:23.146600 (+ 4403us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.115.1:59759: BlockingRecv error: recv got EOF from 127.23.115.1:59759 (error 108)
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":485,"threads_started":1}
Jun 20 18:39:23 dist-test-slave-d645 krb5kdc[32250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.115.3@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:39:23 dist-test-slave-d645 krb5kdc[32250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.115.3@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:39:23.212075 841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:23.201114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:43103 (local address 127.23.115.3:43763)
0620 18:39:23.201584 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:23.201600 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:23.201754 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:23.202522 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:23.202527 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:23.203089 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:23.203294 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:23.203305 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:23.204582 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:23.204587 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:23.205624 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:23.205640 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:23.206471 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:23.206476 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:23.206778 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:23.207493 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:23.207511 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:23.211828 (+ 4317us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.115.62:43103: Server oryx/127.23.115.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":250,"threads_started":1}
W20260620 18:39:23.212604 840 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:23.201224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.3:43763 (local address 127.23.115.62:43103)
0620 18:39:23.201531 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:23.201537 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:23.201695 (+ 158us) server_negotiation.cc:407] Connection header received
0620 18:39:23.202026 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:23.202034 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:23.202151 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:23.202333 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:23.203582 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:23.204367 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:23.205833 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:23.206263 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:23.206373 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:23.212291 (+ 5918us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.115.3:43763: BlockingRecv error: recv got EOF from 127.23.115.3:43763 (error 108)
Metrics: {"server-negotiator.queue_time_us":161}
Jun 20 18:39:23 dist-test-slave-d645 krb5kdc[32250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.115.2@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:39:23 dist-test-slave-d645 krb5kdc[32250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.115.2@KRBTEST.COM for oryx/127.23.115.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:39:23.522166 842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:23.511822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:43103 (local address 127.23.115.2:42901)
0620 18:39:23.512418 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:23.512436 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:23.512578 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:23.513041 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:23.513044 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:23.513390 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:23.513572 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:23.513581 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:23.514705 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:23.514710 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:23.516228 (+ 1518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:23.516243 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:23.517593 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:23.517598 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:23.517775 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:23.518422 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:23.518441 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:23.521928 (+ 3487us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.115.62:43103: Server oryx/127.23.115.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":305,"threads_started":1}
W20260620 18:39:23.522464 840 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:23.511928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.2:42901 (local address 127.23.115.62:43103)
0620 18:39:23.512249 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:23.512257 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:23.512548 (+ 291us) server_negotiation.cc:407] Connection header received
0620 18:39:23.512824 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:23.512828 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:23.512915 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:23.513008 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:23.513771 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:23.514525 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:23.516471 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:23.517373 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:23.517551 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:23.522207 (+ 4656us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.115.2:42901: BlockingRecv error: recv got EOF from 127.23.115.2:42901 (error 108)
Metrics: {"server-negotiator.queue_time_us":124}
I20260620 18:39:24.040870 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 369
I20260620 18:39:24.052446 844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:24.041909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56600 (local address 127.23.115.1:46607)
0620 18:39:24.042479 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:24.042486 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:24.042510 (+ 24us) server_negotiation.cc:407] Connection header received
0620 18:39:24.042574 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:24.042581 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:24.042812 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:24.043008 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:24.044354 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:24.045382 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:24.046594 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:24.047210 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:24.047312 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:24.049583 (+ 2271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:24.049591 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:24.049604 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:24.049639 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:24.051942 (+ 2303us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.23.115.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0620 18:39:24.052236 (+ 294us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56600: Request ticket server kudu/127.23.115.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":272,"threads_started":1}
W20260620 18:39:24.052548 844 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56600: Request ticket server kudu/127.23.115.1@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260620 18:39:24.052599 24012 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 369: Not authorized: Client connection negotiation failed: client connection to 127.23.115.1:46607: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.23.115.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260620 18:39:24.052806 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 369
I20260620 18:39:24.066987 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 520
I20260620 18:39:24.080420 845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:24.068048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45938 (local address 127.23.115.2:45615)
0620 18:39:24.068609 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:24.068616 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:24.068640 (+ 24us) server_negotiation.cc:407] Connection header received
0620 18:39:24.068711 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:24.068717 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:24.068947 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:24.069106 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:24.070704 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:24.071902 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:24.073448 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:24.074006 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:24.074117 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:24.076634 (+ 2517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:24.076644 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:24.076658 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:24.076707 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:24.079894 (+ 3187us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.23.115.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0620 18:39:24.080198 (+ 304us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45938: Request ticket server kudu/127.23.115.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":281,"threads_started":1}
W20260620 18:39:24.080535 845 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45938: Request ticket server kudu/127.23.115.2@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260620 18:39:24.080663 24012 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 520: Not authorized: Client connection negotiation failed: client connection to 127.23.115.2:45615: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.23.115.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260620 18:39:24.080806 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 520
I20260620 18:39:24.094822 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 653
W20260620 18:39:24.107450 24012 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 653: Not authorized: Client connection negotiation failed: client connection to 127.23.115.3:39625: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.23.115.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260620 18:39:24.107605 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 653
I20260620 18:39:24.107615 846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:24.095971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39544 (local address 127.23.115.3:39625)
0620 18:39:24.096653 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:24.096662 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:24.096692 (+ 30us) server_negotiation.cc:407] Connection header received
0620 18:39:24.096776 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:24.096785 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:24.097086 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:24.097290 (+ 204us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:24.098621 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:24.099660 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:24.100865 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:24.101406 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:24.101512 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:24.104081 (+ 2569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:24.104089 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:24.104110 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:24.104151 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:24.106769 (+ 2618us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.23.115.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0620 18:39:24.107276 (+ 507us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39544: Request ticket server kudu/127.23.115.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":477,"thread_start_us":372,"threads_started":1}
I20260620 18:39:24.120509 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 300
I20260620 18:39:24.121682 362 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:39:24.237689 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 300
2026-06-20T18:39:24Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21201 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-20T18:39:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:39:24Z Disabled control of system clock
I20260620 18:39:24.309723 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980676253068-24012-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980676253068-24012-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980676253068-24012-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.115.62:35473
--webserver_interface=127.23.115.62
--webserver_port=0
--builtin_ntp_servers=127.23.115.20:35201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.115.62:35473
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 18:39:24.639490 851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:39:24.639936 851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:39:24.640074 851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:39:24.652415 851 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:39:24.652539 851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:39:24.652575 851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:39:24.652629 851 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:39:24.652670 851 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:39:24.662055 851 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:35201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980676253068-24012-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980676253068-24012-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.115.62:35473
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.115.62:35473
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.115.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980676253068-24012-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:39:24.663403 851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:39:24.665100 851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:39:24.676507 857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:24.676559 859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:24.676563 856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:24.677680 851 server_base.cc:1034] running on GCE node
I20260620 18:39:24.678440 851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:39:24.679872 851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:39:24.681082 851 hybrid_clock.cc:648] HybridClock initialized: now 1781980764681035 us; error 35 us; skew 500 ppm
I20260620 18:39:24.681421 851 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:39:24.683738 851 webserver.cc:460] Webserver started at http://127.23.115.62:33065/ using document root <none> and password file <none>
I20260620 18:39:24.684419 851 fs_manager.cc:359] Metadata directory not provided
I20260620 18:39:24.684514 851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:39:24.684814 851 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:39:24.687675 851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980676253068-24012-0/minicluster-data/master-0/data/instance:
uuid: "3036bfc3d9564cabb5286f31fff95235"
format_stamp: "Formatted at 2026-06-20 18:39:24 on dist-test-slave-d645"
server_key: "d0436b0b9345c494a6e7e2834a52d495"
server_key_iv: "f6bfdeefc6d8208a307e65b4c02440f8"
server_key_version: "encryptionkey@0"
I20260620 18:39:24.688412 851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980676253068-24012-0/minicluster-data/master-0/wal/instance:
uuid: "3036bfc3d9564cabb5286f31fff95235"
format_stamp: "Formatted at 2026-06-20 18:39:24 on dist-test-slave-d645"
server_key: "d0436b0b9345c494a6e7e2834a52d495"
server_key_iv: "f6bfdeefc6d8208a307e65b4c02440f8"
server_key_version: "encryptionkey@0"
I20260620 18:39:24.693400 851 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260620 18:39:24.696941 865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:24.698333 851 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:39:24.698505 851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980676253068-24012-0/minicluster-data/master-0/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980676253068-24012-0/minicluster-data/master-0/wal
uuid: "3036bfc3d9564cabb5286f31fff95235"
format_stamp: "Formatted at 2026-06-20 18:39:24 on dist-test-slave-d645"
server_key: "d0436b0b9345c494a6e7e2834a52d495"
server_key_iv: "f6bfdeefc6d8208a307e65b4c02440f8"
server_key_version: "encryptionkey@0"
I20260620 18:39:24.698717 851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980676253068-24012-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980676253068-24012-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980676253068-24012-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:39:24.714162 851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:39:24.715026 851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:39:24.715298 851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:39:24.732697 851 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.62:35473
I20260620 18:39:24.732717 916 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.62:35473 every 8 connection(s)
I20260620 18:39:24.734758 851 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
I20260620 18:39:24.738831 917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:39:24.739773 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 851
I20260620 18:39:24.739955 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980676253068-24012-0/minicluster-data/master-0/wal/instance
I20260620 18:39:24.740623 24012 external_mini_cluster.cc:1428] Setting key fa694121b96feebe8ccdc8a96078febf
I20260620 18:39:24.752111 917 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3036bfc3d9564cabb5286f31fff95235: Bootstrap starting.
I20260620 18:39:24.754467 920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:24.742721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54944 (local address 127.23.115.62:35473)
0620 18:39:24.743832 (+ 1111us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:24.743843 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:24.743886 (+ 43us) server_negotiation.cc:407] Connection header received
0620 18:39:24.745986 (+ 2100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:24.746010 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:24.746533 (+ 523us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:24.746916 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:24.747585 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:24.748862 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:24.750152 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:24.750850 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:24.751051 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:24.752204 (+ 1153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:24.752211 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:24.752225 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:39:24.752783 (+ 558us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:24.752859 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:39:24.752881 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:24.752979 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:24.753222 (+ 243us) server_negotiation.cc:299] Negotiation successful
0620 18:39:24.753535 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":628,"thread_start_us":363,"threads_started":1}
I20260620 18:39:24.755468 917 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3036bfc3d9564cabb5286f31fff95235: Neither blocks nor log segments found. Creating new log.
I20260620 18:39:24.756560 917 log.cc:824] T 00000000000000000000000000000000 P 3036bfc3d9564cabb5286f31fff95235: Log is configured to *not* fsync() on all Append() calls
I20260620 18:39:24.759245 917 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3036bfc3d9564cabb5286f31fff95235: No bootstrap required, opened a new log
I20260620 18:39:24.767257 917 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3036bfc3d9564cabb5286f31fff95235 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3036bfc3d9564cabb5286f31fff95235" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 35473 } }
I20260620 18:39:24.767663 917 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3036bfc3d9564cabb5286f31fff95235 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:39:24.767755 917 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3036bfc3d9564cabb5286f31fff95235 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3036bfc3d9564cabb5286f31fff95235, State: Initialized, Role: FOLLOWER
I20260620 18:39:24.768311 917 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3036bfc3d9564cabb5286f31fff95235 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3036bfc3d9564cabb5286f31fff95235" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 35473 } }
I20260620 18:39:24.768508 917 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3036bfc3d9564cabb5286f31fff95235 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:39:24.768610 917 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3036bfc3d9564cabb5286f31fff95235 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:39:24.768724 917 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3036bfc3d9564cabb5286f31fff95235 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:39:24.770836 917 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3036bfc3d9564cabb5286f31fff95235 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3036bfc3d9564cabb5286f31fff95235" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 35473 } }
I20260620 18:39:24.771343 917 leader_election.cc:302] T 00000000000000000000000000000000 P 3036bfc3d9564cabb5286f31fff95235 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3036bfc3d9564cabb5286f31fff95235; no voters:
I20260620 18:39:24.771862 917 leader_election.cc:288] T 00000000000000000000000000000000 P 3036bfc3d9564cabb5286f31fff95235 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:39:24.771993 922 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3036bfc3d9564cabb5286f31fff95235 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:39:24.772663 922 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3036bfc3d9564cabb5286f31fff95235 [term 1 LEADER]: Becoming Leader. State: Replica: 3036bfc3d9564cabb5286f31fff95235, State: Running, Role: LEADER
I20260620 18:39:24.773166 922 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3036bfc3d9564cabb5286f31fff95235 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3036bfc3d9564cabb5286f31fff95235" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 35473 } }
I20260620 18:39:24.773830 917 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3036bfc3d9564cabb5286f31fff95235 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:39:24.776389 924 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3036bfc3d9564cabb5286f31fff95235 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3036bfc3d9564cabb5286f31fff95235. Latest consensus state: current_term: 1 leader_uuid: "3036bfc3d9564cabb5286f31fff95235" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3036bfc3d9564cabb5286f31fff95235" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 35473 } } }
I20260620 18:39:24.776796 924 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3036bfc3d9564cabb5286f31fff95235 [sys.catalog]: This master's current role is: LEADER
I20260620 18:39:24.776343 923 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3036bfc3d9564cabb5286f31fff95235 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3036bfc3d9564cabb5286f31fff95235" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3036bfc3d9564cabb5286f31fff95235" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 35473 } } }
I20260620 18:39:24.776969 923 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3036bfc3d9564cabb5286f31fff95235 [sys.catalog]: This master's current role is: LEADER
I20260620 18:39:24.778800 929 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:39:24.784955 929 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:39:24.794394 929 catalog_manager.cc:1347] Generated new cluster ID: c5d1edc8ba174d3e854b839901c653f3
I20260620 18:39:24.794518 929 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:39:24.805410 929 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:39:24.807224 929 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:39:24.818091 929 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3036bfc3d9564cabb5286f31fff95235: Generated new TSK 0
I20260620 18:39:24.819211 929 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:39:24.824740 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980676253068-24012-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980676253068-24012-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980676253068-24012-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980676253068-24012-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.115.1:0
--local_ip_for_outbound_sockets=127.23.115.1
--webserver_interface=127.23.115.1
--webserver_port=0
--tserver_master_addrs=127.23.115.62:35473
--builtin_ntp_servers=127.23.115.20:35201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260620 18:39:25.149243 941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:39:25.149533 941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:39:25.149642 941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:39:25.156901 941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:39:25.157039 941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:39:25.157188 941 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.115.1
I20260620 18:39:25.166209 941 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:35201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980676253068-24012-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980676253068-24012-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.115.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980676253068-24012-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.115.1
--webserver_port=0
--tserver_master_addrs=127.23.115.62:35473
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.115.1
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980676253068-24012-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:39:25.167467 941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:39:25.169194 941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:39:25.182734 946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:25.182725 947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:25.182783 949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:25.183687 941 server_base.cc:1034] running on GCE node
I20260620 18:39:25.184303 941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:39:25.185281 941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:39:25.186497 941 hybrid_clock.cc:648] HybridClock initialized: now 1781980765186465 us; error 27 us; skew 500 ppm
I20260620 18:39:25.186825 941 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:39:25.194096 941 webserver.cc:460] Webserver started at http://127.23.115.1:46497/ using document root <none> and password file <none>
I20260620 18:39:25.194898 941 fs_manager.cc:359] Metadata directory not provided
I20260620 18:39:25.195050 941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:39:25.195367 941 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:39:25.198312 941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980676253068-24012-0/minicluster-data/ts-0/data/instance:
uuid: "e09db47c7c4548fbbb16ddaa59e939eb"
format_stamp: "Formatted at 2026-06-20 18:39:25 on dist-test-slave-d645"
server_key: "9b9f62ad6a3fc5d9ab2666f04126440d"
server_key_iv: "49a40327108fdedae910d7922032cfca"
server_key_version: "encryptionkey@0"
I20260620 18:39:25.199151 941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980676253068-24012-0/minicluster-data/ts-0/wal/instance:
uuid: "e09db47c7c4548fbbb16ddaa59e939eb"
format_stamp: "Formatted at 2026-06-20 18:39:25 on dist-test-slave-d645"
server_key: "9b9f62ad6a3fc5d9ab2666f04126440d"
server_key_iv: "49a40327108fdedae910d7922032cfca"
server_key_version: "encryptionkey@0"
I20260620 18:39:25.204434 941 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260620 18:39:25.208264 955 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:25.209982 941 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 18:39:25.210160 941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980676253068-24012-0/minicluster-data/ts-0/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980676253068-24012-0/minicluster-data/ts-0/wal
uuid: "e09db47c7c4548fbbb16ddaa59e939eb"
format_stamp: "Formatted at 2026-06-20 18:39:25 on dist-test-slave-d645"
server_key: "9b9f62ad6a3fc5d9ab2666f04126440d"
server_key_iv: "49a40327108fdedae910d7922032cfca"
server_key_version: "encryptionkey@0"
I20260620 18:39:25.210364 941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980676253068-24012-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980676253068-24012-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980676253068-24012-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:39:25.228888 941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:39:25.229718 941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:39:25.229986 941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:39:25.230916 941 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:39:25.232429 941 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:39:25.232512 941 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:25.232614 941 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:39:25.232688 941 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:25.263906 941 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.1:46677
I20260620 18:39:25.263954 1067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.1:46677 every 8 connection(s)
I20260620 18:39:25.265187 941 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980676253068-24012-0/minicluster-data/ts-0/data/info.pb
I20260620 18:39:25.266868 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 941
I20260620 18:39:25.267079 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980676253068-24012-0/minicluster-data/ts-0/wal/instance
I20260620 18:39:25.268011 24012 external_mini_cluster.cc:1428] Setting key b1b548874015eff3810c4cda6b0c6e27
I20260620 18:39:25.271543 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980676253068-24012-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980676253068-24012-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980676253068-24012-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980676253068-24012-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.115.2:0
--local_ip_for_outbound_sockets=127.23.115.2
--webserver_interface=127.23.115.2
--webserver_port=0
--tserver_master_addrs=127.23.115.62:35473
--builtin_ntp_servers=127.23.115.20:35201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 18:39:25.283179 1070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:25.268678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.1:48459 (local address 127.23.115.62:35473)
0620 18:39:25.269171 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:25.269177 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:25.270735 (+ 1558us) server_negotiation.cc:407] Connection header received
0620 18:39:25.273406 (+ 2671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:25.273412 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:25.273553 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:25.273698 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:25.275307 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:25.276292 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:25.279144 (+ 2852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:25.279854 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:25.280009 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:25.282065 (+ 2056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:25.282072 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:25.282076 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:39:25.282108 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:25.282180 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:39:25.282199 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:25.282377 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:25.282749 (+ 372us) server_negotiation.cc:299] Negotiation successful
0620 18:39:25.282849 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":241,"threads_started":1}
I20260620 18:39:25.284420 1071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:25.269306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:35473 (local address 127.23.115.1:48459)
0620 18:39:25.270538 (+ 1232us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:25.270595 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:25.272927 (+ 2332us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:25.273968 (+ 1041us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:25.273979 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:25.274093 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:25.274919 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:25.274968 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:25.276622 (+ 1654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:25.276630 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:25.278862 (+ 2232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:25.278906 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:25.280301 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:25.280426 (+ 125us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:25.280757 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:25.281513 (+ 756us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:39:25.281517 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:25.281570 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:39:25.281677 (+ 107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:39:25.282393 (+ 716us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:39:25.282403 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:39:25.282411 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 18:39:25.282615 (+ 204us) client_negotiation.cc:240] Negotiation successful
0620 18:39:25.283056 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":669,"thread_start_us":431,"threads_started":1}
I20260620 18:39:25.286141 1068 heartbeater.cc:347] Connected to a master server at 127.23.115.62:35473
I20260620 18:39:25.286465 1068 heartbeater.cc:464] Registering TS with master...
I20260620 18:39:25.287384 1068 heartbeater.cc:511] Master 127.23.115.62:35473 requested a full tablet report, sending...
I20260620 18:39:25.289649 882 ts_manager.cc:194] Registered new tserver with Master: e09db47c7c4548fbbb16ddaa59e939eb (127.23.115.1:46677)
I20260620 18:39:25.291730 882 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.115.1:48459
W20260620 18:39:25.600688 1072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:39:25.600965 1072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:39:25.601048 1072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:39:25.608340 1072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:39:25.608439 1072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:39:25.608542 1072 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.115.2
I20260620 18:39:25.617748 1072 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:35201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980676253068-24012-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980676253068-24012-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.115.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980676253068-24012-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.115.2
--webserver_port=0
--tserver_master_addrs=127.23.115.62:35473
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.115.2
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980676253068-24012-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:39:25.618921 1072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:39:25.620622 1072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:39:25.632988 1078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:25.633057 1080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:25.633025 1077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:25.633720 1072 server_base.cc:1034] running on GCE node
I20260620 18:39:25.634280 1072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:39:25.635195 1072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:39:25.636401 1072 hybrid_clock.cc:648] HybridClock initialized: now 1781980765636357 us; error 40 us; skew 500 ppm
I20260620 18:39:25.636731 1072 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:39:25.639217 1072 webserver.cc:460] Webserver started at http://127.23.115.2:43315/ using document root <none> and password file <none>
I20260620 18:39:25.639904 1072 fs_manager.cc:359] Metadata directory not provided
I20260620 18:39:25.640059 1072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:39:25.640471 1072 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:39:25.643364 1072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980676253068-24012-0/minicluster-data/ts-1/data/instance:
uuid: "feede7e111c6413faed985d337d1a3a1"
format_stamp: "Formatted at 2026-06-20 18:39:25 on dist-test-slave-d645"
server_key: "86cdd79b0fa1e3eefa9a615f7e704f6f"
server_key_iv: "da91b221b85cf632ed5f51f47fc2a5c1"
server_key_version: "encryptionkey@0"
I20260620 18:39:25.644152 1072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980676253068-24012-0/minicluster-data/ts-1/wal/instance:
uuid: "feede7e111c6413faed985d337d1a3a1"
format_stamp: "Formatted at 2026-06-20 18:39:25 on dist-test-slave-d645"
server_key: "86cdd79b0fa1e3eefa9a615f7e704f6f"
server_key_iv: "da91b221b85cf632ed5f51f47fc2a5c1"
server_key_version: "encryptionkey@0"
I20260620 18:39:25.649228 1072 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.005s
I20260620 18:39:25.653110 1086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:25.654877 1072 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:39:25.655083 1072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980676253068-24012-0/minicluster-data/ts-1/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980676253068-24012-0/minicluster-data/ts-1/wal
uuid: "feede7e111c6413faed985d337d1a3a1"
format_stamp: "Formatted at 2026-06-20 18:39:25 on dist-test-slave-d645"
server_key: "86cdd79b0fa1e3eefa9a615f7e704f6f"
server_key_iv: "da91b221b85cf632ed5f51f47fc2a5c1"
server_key_version: "encryptionkey@0"
I20260620 18:39:25.655303 1072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980676253068-24012-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980676253068-24012-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980676253068-24012-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:39:25.686797 1072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:39:25.687740 1072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:39:25.688045 1072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:39:25.689054 1072 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:39:25.690441 1072 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:39:25.690527 1072 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:25.690609 1072 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:39:25.690668 1072 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:25.721760 1072 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.2:46179
I20260620 18:39:25.722070 1198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.2:46179 every 8 connection(s)
I20260620 18:39:25.723073 1072 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980676253068-24012-0/minicluster-data/ts-1/data/info.pb
I20260620 18:39:25.729182 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 1072
I20260620 18:39:25.729377 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980676253068-24012-0/minicluster-data/ts-1/wal/instance
I20260620 18:39:25.729890 24012 external_mini_cluster.cc:1428] Setting key ace7fdb1258bc9c4d0b04b75545a6545
I20260620 18:39:25.733048 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980676253068-24012-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980676253068-24012-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980676253068-24012-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980676253068-24012-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.115.3:0
--local_ip_for_outbound_sockets=127.23.115.3
--webserver_interface=127.23.115.3
--webserver_port=0
--tserver_master_addrs=127.23.115.62:35473
--builtin_ntp_servers=127.23.115.20:35201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 18:39:25.736577 1070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:25.725508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.2:50639 (local address 127.23.115.62:35473)
0620 18:39:25.725709 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:25.725723 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:25.727794 (+ 2071us) server_negotiation.cc:407] Connection header received
0620 18:39:25.729785 (+ 1991us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:25.729789 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:25.729863 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:25.729960 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:25.731221 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:25.732169 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:25.733905 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:25.734326 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:25.734432 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:25.735742 (+ 1310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:25.735748 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:25.735751 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:39:25.735769 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:25.735803 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:39:25.735815 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:25.735889 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:25.736106 (+ 217us) server_negotiation.cc:299] Negotiation successful
0620 18:39:25.736210 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":62}
I20260620 18:39:25.737571 1201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:25.726081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:35473 (local address 127.23.115.2:50639)
0620 18:39:25.727579 (+ 1498us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:25.727618 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:25.729468 (+ 1850us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:25.730224 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:25.730236 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:25.730371 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:25.731050 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:25.731069 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:25.732375 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:25.732379 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:25.733767 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:25.733780 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:25.734532 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:25.734539 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:25.734762 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:25.735435 (+ 673us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:39:25.735437 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:25.735475 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:39:25.735551 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:39:25.735897 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:39:25.735908 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:39:25.735916 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 18:39:25.736061 (+ 145us) client_negotiation.cc:240] Negotiation successful
0620 18:39:25.736450 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":635,"thread_start_us":386,"threads_started":1}
I20260620 18:39:25.738854 1199 heartbeater.cc:347] Connected to a master server at 127.23.115.62:35473
I20260620 18:39:25.739149 1199 heartbeater.cc:464] Registering TS with master...
I20260620 18:39:25.739874 1199 heartbeater.cc:511] Master 127.23.115.62:35473 requested a full tablet report, sending...
I20260620 18:39:25.741191 882 ts_manager.cc:194] Registered new tserver with Master: feede7e111c6413faed985d337d1a3a1 (127.23.115.2:46179)
I20260620 18:39:25.742233 882 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.115.2:50639
W20260620 18:39:26.059269 1202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:39:26.059540 1202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:39:26.059646 1202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:39:26.067126 1202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:39:26.067233 1202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:39:26.067389 1202 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.115.3
I20260620 18:39:26.076695 1202 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:35201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980676253068-24012-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980676253068-24012-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.115.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980676253068-24012-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.115.3
--webserver_port=0
--tserver_master_addrs=127.23.115.62:35473
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.115.3
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980676253068-24012-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:39:26.077944 1202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:39:26.079829 1202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:39:26.092571 1208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:26.092617 1207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:26.092622 1210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:26.093513 1202 server_base.cc:1034] running on GCE node
I20260620 18:39:26.094100 1202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:39:26.095054 1202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:39:26.096230 1202 hybrid_clock.cc:648] HybridClock initialized: now 1781980766096183 us; error 45 us; skew 500 ppm
I20260620 18:39:26.096511 1202 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:39:26.098711 1202 webserver.cc:460] Webserver started at http://127.23.115.3:45527/ using document root <none> and password file <none>
I20260620 18:39:26.099427 1202 fs_manager.cc:359] Metadata directory not provided
I20260620 18:39:26.099514 1202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:39:26.099812 1202 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:39:26.102586 1202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980676253068-24012-0/minicluster-data/ts-2/data/instance:
uuid: "cc289903c0bd4ce1b2f2226bd4e9ff6d"
format_stamp: "Formatted at 2026-06-20 18:39:26 on dist-test-slave-d645"
server_key: "4bf03382034013f9e5dc6f979c207236"
server_key_iv: "c34f177ba59bec60f715c824453c7402"
server_key_version: "encryptionkey@0"
I20260620 18:39:26.103461 1202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980676253068-24012-0/minicluster-data/ts-2/wal/instance:
uuid: "cc289903c0bd4ce1b2f2226bd4e9ff6d"
format_stamp: "Formatted at 2026-06-20 18:39:26 on dist-test-slave-d645"
server_key: "4bf03382034013f9e5dc6f979c207236"
server_key_iv: "c34f177ba59bec60f715c824453c7402"
server_key_version: "encryptionkey@0"
I20260620 18:39:26.108338 1202 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260620 18:39:26.111682 1216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:26.112962 1202 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:39:26.113111 1202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980676253068-24012-0/minicluster-data/ts-2/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980676253068-24012-0/minicluster-data/ts-2/wal
uuid: "cc289903c0bd4ce1b2f2226bd4e9ff6d"
format_stamp: "Formatted at 2026-06-20 18:39:26 on dist-test-slave-d645"
server_key: "4bf03382034013f9e5dc6f979c207236"
server_key_iv: "c34f177ba59bec60f715c824453c7402"
server_key_version: "encryptionkey@0"
I20260620 18:39:26.113323 1202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980676253068-24012-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980676253068-24012-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980676253068-24012-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:39:26.124356 1202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:39:26.125144 1202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:39:26.125412 1202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:39:26.126322 1202 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:39:26.127784 1202 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:39:26.127876 1202 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:26.127977 1202 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:39:26.128054 1202 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:26.157862 1202 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.3:37265
I20260620 18:39:26.157927 1328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.3:37265 every 8 connection(s)
I20260620 18:39:26.159170 1202 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980676253068-24012-0/minicluster-data/ts-2/data/info.pb
I20260620 18:39:26.160851 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 1202
I20260620 18:39:26.161011 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980676253068-24012-0/minicluster-data/ts-2/wal/instance
I20260620 18:39:26.161485 24012 external_mini_cluster.cc:1428] Setting key 61da19a8296a39d3cff645bdb60a581c
I20260620 18:39:26.173461 1070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:26.162227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.3:50891 (local address 127.23.115.62:35473)
0620 18:39:26.162511 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:26.162517 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:26.164246 (+ 1729us) server_negotiation.cc:407] Connection header received
0620 18:39:26.166064 (+ 1818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:26.166068 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:26.166133 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:26.166234 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:26.167374 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:26.168197 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:26.169785 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:26.170235 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:26.170347 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:26.172550 (+ 2203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:26.172556 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:26.172560 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:39:26.172580 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:26.172617 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:39:26.172631 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:26.172719 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:26.172994 (+ 275us) server_negotiation.cc:299] Negotiation successful
0620 18:39:26.173101 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137}
I20260620 18:39:26.174635 1331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:26.162925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:35473 (local address 127.23.115.3:50891)
0620 18:39:26.164123 (+ 1198us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:26.164164 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:26.165805 (+ 1641us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:26.166460 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:26.166468 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:26.166558 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:26.167187 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:26.167207 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:26.168369 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:26.168373 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:26.169641 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:26.169654 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:26.170613 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:26.170630 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:26.171019 (+ 389us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:26.172053 (+ 1034us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:39:26.172056 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:26.172116 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:39:26.172233 (+ 117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:39:26.172743 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:39:26.172756 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:39:26.172763 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 18:39:26.172951 (+ 188us) client_negotiation.cc:240] Negotiation successful
0620 18:39:26.173403 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":412,"threads_started":1}
I20260620 18:39:26.176312 1329 heartbeater.cc:347] Connected to a master server at 127.23.115.62:35473
I20260620 18:39:26.176599 1329 heartbeater.cc:464] Registering TS with master...
I20260620 18:39:26.177534 1329 heartbeater.cc:511] Master 127.23.115.62:35473 requested a full tablet report, sending...
I20260620 18:39:26.178891 882 ts_manager.cc:194] Registered new tserver with Master: cc289903c0bd4ce1b2f2226bd4e9ff6d (127.23.115.3:37265)
I20260620 18:39:26.179718 882 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.115.3:50891
I20260620 18:39:26.189741 24012 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 18:39:26.195986 1070 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:26.194495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54954 (local address 127.23.115.62:35473)
0620 18:39:26.194706 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:26.194711 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:26.195200 (+ 489us) server_negotiation.cc:407] Connection header received
0620 18:39:26.195328 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:26.195332 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:26.195408 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:26.195492 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:26.195814 (+ 322us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54954: BlockingRecv error: recv got EOF from 127.0.0.1:54954 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 18:39:26.196645 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 941
I20260620 18:39:26.204432 1338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:26.197522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56124 (local address 127.23.115.1:46677)
0620 18:39:26.198013 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:26.198020 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:26.198043 (+ 23us) server_negotiation.cc:407] Connection header received
0620 18:39:26.198093 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:26.198099 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:26.198298 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:26.198428 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:26.198794 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:26.199966 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:26.201205 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:26.201785 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:26.201898 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:26.202977 (+ 1079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:26.202988 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:26.203005 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:39:26.203623 (+ 618us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:26.203682 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:39:26.203704 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:26.203803 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:26.203942 (+ 139us) server_negotiation.cc:299] Negotiation successful
0620 18:39:26.204087 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":220,"threads_started":1}
I20260620 18:39:26.205651 1063 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:39:26.288220 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 941
I20260620 18:39:26.301630 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 1072
I20260620 18:39:26.309630 1340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:26.302498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38918 (local address 127.23.115.2:46179)
0620 18:39:26.303060 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:26.303067 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:26.303092 (+ 25us) server_negotiation.cc:407] Connection header received
0620 18:39:26.303166 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:26.303172 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:26.303387 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:26.303536 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:26.303953 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:26.304996 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:26.306235 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:26.306835 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:26.306972 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:26.307975 (+ 1003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:26.307988 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:26.308011 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:39:26.308597 (+ 586us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:26.308654 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:39:26.308687 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:26.308781 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:26.309007 (+ 226us) server_negotiation.cc:299] Negotiation successful
0620 18:39:26.309218 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":307,"threads_started":1}
I20260620 18:39:26.311143 1194 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:39:26.392009 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 1072
I20260620 18:39:26.405401 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 1202
I20260620 18:39:26.414467 1342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:26.406397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54282 (local address 127.23.115.3:37265)
0620 18:39:26.407406 (+ 1009us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:26.407416 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:26.407461 (+ 45us) server_negotiation.cc:407] Connection header received
0620 18:39:26.407552 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:26.407560 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:26.407866 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:26.408059 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:26.408616 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:26.409835 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:26.411260 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:26.411906 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:26.412011 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:26.413039 (+ 1028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:26.413047 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:26.413060 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:39:26.413666 (+ 606us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:26.413712 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:39:26.413731 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:26.413839 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:26.414005 (+ 166us) server_negotiation.cc:299] Negotiation successful
0620 18:39:26.414144 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":731,"thread_start_us":406,"threads_started":1}
I20260620 18:39:26.416119 1324 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:39:26.500484 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 1202
I20260620 18:39:26.514369 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 851
I20260620 18:39:26.515655 912 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:39:26.619988 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 851
2026-06-20T18:39:26Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2380 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-20T18:39:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:39:26Z Disabled control of system clock
I20260620 18:39:26.691351 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980676253068-24012-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980676253068-24012-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980676253068-24012-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.115.62:35451
--webserver_interface=127.23.115.62
--webserver_port=0
--builtin_ntp_servers=127.23.115.20:44057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.115.62:35451
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 18:39:27.035645 1348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:39:27.035925 1348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:39:27.036013 1348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:39:27.043092 1348 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:39:27.043195 1348 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:39:27.043231 1348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:39:27.043300 1348 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:39:27.043342 1348 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:39:27.052356 1348 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:44057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980676253068-24012-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980676253068-24012-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.115.62:35451
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.115.62:35451
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.115.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980676253068-24012-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:39:27.053546 1348 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:39:27.055289 1348 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:39:27.067695 1353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:27.067795 1356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:27.067838 1354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:27.072901 1348 server_base.cc:1034] running on GCE node
I20260620 18:39:27.073773 1348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:39:27.075307 1348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:39:27.076561 1348 hybrid_clock.cc:648] HybridClock initialized: now 1781980767076502 us; error 76 us; skew 500 ppm
I20260620 18:39:27.076902 1348 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:39:27.079265 1348 webserver.cc:460] Webserver started at http://127.23.115.62:37473/ using document root <none> and password file <none>
I20260620 18:39:27.080003 1348 fs_manager.cc:359] Metadata directory not provided
I20260620 18:39:27.080101 1348 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:39:27.080417 1348 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:39:27.083362 1348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980676253068-24012-0/minicluster-data/master-0/data/instance:
uuid: "925c40d7f7c34ad98c632b282761aee7"
format_stamp: "Formatted at 2026-06-20 18:39:27 on dist-test-slave-d645"
server_key: "bc67ec144ac8cc604dc28a998e360e50"
server_key_iv: "28dcb06ce6e8cea60fc9f895c9834e1f"
server_key_version: "encryptionkey@0"
I20260620 18:39:27.084142 1348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980676253068-24012-0/minicluster-data/master-0/wal/instance:
uuid: "925c40d7f7c34ad98c632b282761aee7"
format_stamp: "Formatted at 2026-06-20 18:39:27 on dist-test-slave-d645"
server_key: "bc67ec144ac8cc604dc28a998e360e50"
server_key_iv: "28dcb06ce6e8cea60fc9f895c9834e1f"
server_key_version: "encryptionkey@0"
I20260620 18:39:27.089246 1348 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:39:27.093581 1362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:27.095146 1348 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 18:39:27.095302 1348 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980676253068-24012-0/minicluster-data/master-0/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980676253068-24012-0/minicluster-data/master-0/wal
uuid: "925c40d7f7c34ad98c632b282761aee7"
format_stamp: "Formatted at 2026-06-20 18:39:27 on dist-test-slave-d645"
server_key: "bc67ec144ac8cc604dc28a998e360e50"
server_key_iv: "28dcb06ce6e8cea60fc9f895c9834e1f"
server_key_version: "encryptionkey@0"
I20260620 18:39:27.095504 1348 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980676253068-24012-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980676253068-24012-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980676253068-24012-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:39:27.101886 1348 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:39:27.102784 1348 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:39:27.103138 1348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:39:27.122129 1348 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.62:35451
I20260620 18:39:27.122193 1413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.62:35451 every 8 connection(s)
I20260620 18:39:27.123503 1348 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
I20260620 18:39:27.127194 1414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:39:27.132129 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 1348
I20260620 18:39:27.132344 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980676253068-24012-0/minicluster-data/master-0/wal/instance
I20260620 18:39:27.133116 24012 external_mini_cluster.cc:1428] Setting key 964dc63e60e2e64a67e8a0b3a41c247a
I20260620 18:39:27.138767 1414 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 925c40d7f7c34ad98c632b282761aee7: Bootstrap starting.
I20260620 18:39:27.141189 1417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:27.135184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45542 (local address 127.23.115.62:35451)
0620 18:39:27.136179 (+ 995us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:27.136192 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:27.136236 (+ 44us) server_negotiation.cc:407] Connection header received
0620 18:39:27.137899 (+ 1663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:27.137920 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:27.138300 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:27.138608 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:27.139691 (+ 1083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:27.139697 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:27.139747 (+ 50us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:39:27.139771 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:27.139826 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:39:27.139843 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:27.140011 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:27.140105 (+ 94us) server_negotiation.cc:299] Negotiation successful
0620 18:39:27.140332 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":384,"threads_started":1}
I20260620 18:39:27.142138 1414 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 925c40d7f7c34ad98c632b282761aee7: Neither blocks nor log segments found. Creating new log.
I20260620 18:39:27.143160 1414 log.cc:824] T 00000000000000000000000000000000 P 925c40d7f7c34ad98c632b282761aee7: Log is configured to *not* fsync() on all Append() calls
I20260620 18:39:27.145803 1414 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 925c40d7f7c34ad98c632b282761aee7: No bootstrap required, opened a new log
I20260620 18:39:27.152630 1414 raft_consensus.cc:348] T 00000000000000000000000000000000 P 925c40d7f7c34ad98c632b282761aee7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "925c40d7f7c34ad98c632b282761aee7" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 35451 } }
I20260620 18:39:27.152963 1414 raft_consensus.cc:374] T 00000000000000000000000000000000 P 925c40d7f7c34ad98c632b282761aee7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:39:27.153048 1414 raft_consensus.cc:729] T 00000000000000000000000000000000 P 925c40d7f7c34ad98c632b282761aee7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 925c40d7f7c34ad98c632b282761aee7, State: Initialized, Role: FOLLOWER
I20260620 18:39:27.153590 1414 consensus_queue.cc:260] T 00000000000000000000000000000000 P 925c40d7f7c34ad98c632b282761aee7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "925c40d7f7c34ad98c632b282761aee7" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 35451 } }
I20260620 18:39:27.153802 1414 raft_consensus.cc:388] T 00000000000000000000000000000000 P 925c40d7f7c34ad98c632b282761aee7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:39:27.153919 1414 raft_consensus.cc:482] T 00000000000000000000000000000000 P 925c40d7f7c34ad98c632b282761aee7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:39:27.154057 1414 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 925c40d7f7c34ad98c632b282761aee7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:39:27.156173 1414 raft_consensus.cc:504] T 00000000000000000000000000000000 P 925c40d7f7c34ad98c632b282761aee7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "925c40d7f7c34ad98c632b282761aee7" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 35451 } }
I20260620 18:39:27.156667 1414 leader_election.cc:302] T 00000000000000000000000000000000 P 925c40d7f7c34ad98c632b282761aee7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 925c40d7f7c34ad98c632b282761aee7; no voters:
I20260620 18:39:27.157176 1414 leader_election.cc:288] T 00000000000000000000000000000000 P 925c40d7f7c34ad98c632b282761aee7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:39:27.157325 1419 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 925c40d7f7c34ad98c632b282761aee7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:39:27.157967 1419 raft_consensus.cc:686] T 00000000000000000000000000000000 P 925c40d7f7c34ad98c632b282761aee7 [term 1 LEADER]: Becoming Leader. State: Replica: 925c40d7f7c34ad98c632b282761aee7, State: Running, Role: LEADER
I20260620 18:39:27.158510 1419 consensus_queue.cc:237] T 00000000000000000000000000000000 P 925c40d7f7c34ad98c632b282761aee7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "925c40d7f7c34ad98c632b282761aee7" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 35451 } }
I20260620 18:39:27.159072 1414 sys_catalog.cc:564] T 00000000000000000000000000000000 P 925c40d7f7c34ad98c632b282761aee7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:39:27.161952 1421 sys_catalog.cc:455] T 00000000000000000000000000000000 P 925c40d7f7c34ad98c632b282761aee7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 925c40d7f7c34ad98c632b282761aee7. Latest consensus state: current_term: 1 leader_uuid: "925c40d7f7c34ad98c632b282761aee7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "925c40d7f7c34ad98c632b282761aee7" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 35451 } } }
I20260620 18:39:27.161958 1420 sys_catalog.cc:455] T 00000000000000000000000000000000 P 925c40d7f7c34ad98c632b282761aee7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "925c40d7f7c34ad98c632b282761aee7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "925c40d7f7c34ad98c632b282761aee7" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 35451 } } }
I20260620 18:39:27.162460 1421 sys_catalog.cc:458] T 00000000000000000000000000000000 P 925c40d7f7c34ad98c632b282761aee7 [sys.catalog]: This master's current role is: LEADER
I20260620 18:39:27.162539 1420 sys_catalog.cc:458] T 00000000000000000000000000000000 P 925c40d7f7c34ad98c632b282761aee7 [sys.catalog]: This master's current role is: LEADER
I20260620 18:39:27.163630 1428 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:39:27.168262 1428 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:39:27.176614 1428 catalog_manager.cc:1347] Generated new cluster ID: 6e4a0506974147028d69a5e2420ef029
I20260620 18:39:27.176733 1428 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:39:27.211150 1428 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:39:27.211656 1428 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:39:27.221859 1428 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 925c40d7f7c34ad98c632b282761aee7: Generated new TSK 0
I20260620 18:39:27.222882 1428 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:39:27.228122 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980676253068-24012-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980676253068-24012-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980676253068-24012-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980676253068-24012-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.115.1:0
--local_ip_for_outbound_sockets=127.23.115.1
--webserver_interface=127.23.115.1
--webserver_port=0
--tserver_master_addrs=127.23.115.62:35451
--builtin_ntp_servers=127.23.115.20:44057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 18:39:27.562429 1438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:39:27.562726 1438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:39:27.562847 1438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:39:27.570115 1438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:39:27.570209 1438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:39:27.570350 1438 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.115.1
I20260620 18:39:27.579489 1438 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:44057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980676253068-24012-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980676253068-24012-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.115.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980676253068-24012-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.115.1
--webserver_port=0
--tserver_master_addrs=127.23.115.62:35451
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.115.1
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980676253068-24012-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:39:27.580893 1438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:39:27.582669 1438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:39:27.596192 1443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:27.596169 1444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:27.596834 1446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:27.596966 1438 server_base.cc:1034] running on GCE node
I20260620 18:39:27.597579 1438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:39:27.598568 1438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:39:27.599800 1438 hybrid_clock.cc:648] HybridClock initialized: now 1781980767599657 us; error 136 us; skew 500 ppm
I20260620 18:39:27.600126 1438 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:39:27.602514 1438 webserver.cc:460] Webserver started at http://127.23.115.1:43105/ using document root <none> and password file <none>
I20260620 18:39:27.603411 1438 fs_manager.cc:359] Metadata directory not provided
I20260620 18:39:27.603534 1438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:39:27.603873 1438 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:39:27.608624 1438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980676253068-24012-0/minicluster-data/ts-0/data/instance:
uuid: "2dd940aad1a541cab7d99929d528d22e"
format_stamp: "Formatted at 2026-06-20 18:39:27 on dist-test-slave-d645"
server_key: "afc49b28bf69d2f55338b8f7e1d2487b"
server_key_iv: "de0d34f3181aa66efcdeb1ed71f477f6"
server_key_version: "encryptionkey@0"
I20260620 18:39:27.609474 1438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980676253068-24012-0/minicluster-data/ts-0/wal/instance:
uuid: "2dd940aad1a541cab7d99929d528d22e"
format_stamp: "Formatted at 2026-06-20 18:39:27 on dist-test-slave-d645"
server_key: "afc49b28bf69d2f55338b8f7e1d2487b"
server_key_iv: "de0d34f3181aa66efcdeb1ed71f477f6"
server_key_version: "encryptionkey@0"
I20260620 18:39:27.614780 1438 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 18:39:27.618280 1452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:27.619566 1438 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:39:27.619711 1438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980676253068-24012-0/minicluster-data/ts-0/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980676253068-24012-0/minicluster-data/ts-0/wal
uuid: "2dd940aad1a541cab7d99929d528d22e"
format_stamp: "Formatted at 2026-06-20 18:39:27 on dist-test-slave-d645"
server_key: "afc49b28bf69d2f55338b8f7e1d2487b"
server_key_iv: "de0d34f3181aa66efcdeb1ed71f477f6"
server_key_version: "encryptionkey@0"
I20260620 18:39:27.619906 1438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980676253068-24012-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980676253068-24012-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980676253068-24012-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:39:27.625303 1438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:39:27.626060 1438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:39:27.626350 1438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:39:27.627460 1438 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:39:27.628891 1438 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:39:27.628985 1438 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:27.629094 1438 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:39:27.629173 1438 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:27.660884 1438 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.1:33833
I20260620 18:39:27.660925 1564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.1:33833 every 8 connection(s)
I20260620 18:39:27.662181 1438 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980676253068-24012-0/minicluster-data/ts-0/data/info.pb
I20260620 18:39:27.662966 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 1438
I20260620 18:39:27.663182 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980676253068-24012-0/minicluster-data/ts-0/wal/instance
I20260620 18:39:27.663993 24012 external_mini_cluster.cc:1428] Setting key 85eeb1029543f8df791292ddcbf86251
I20260620 18:39:27.667302 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980676253068-24012-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980676253068-24012-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980676253068-24012-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980676253068-24012-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.115.2:0
--local_ip_for_outbound_sockets=127.23.115.2
--webserver_interface=127.23.115.2
--webserver_port=0
--tserver_master_addrs=127.23.115.62:35451
--builtin_ntp_servers=127.23.115.20:44057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260620 18:39:27.671470 1567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:27.665375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.1:34749 (local address 127.23.115.62:35451)
0620 18:39:27.665825 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:27.665832 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:27.667150 (+ 1318us) server_negotiation.cc:407] Connection header received
0620 18:39:27.669279 (+ 2129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:27.669284 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:27.669352 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:27.669478 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:27.670734 (+ 1256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:27.670741 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:27.670746 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:39:27.670764 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:27.670804 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:39:27.670815 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:27.670922 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:27.671116 (+ 194us) server_negotiation.cc:299] Negotiation successful
0620 18:39:27.671184 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":213,"threads_started":1}
I20260620 18:39:27.672356 1568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:27.665909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:35451 (local address 127.23.115.1:34749)
0620 18:39:27.667011 (+ 1102us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:27.667059 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:27.668998 (+ 1939us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:27.669712 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:27.669721 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:27.669811 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:27.670446 (+ 635us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:39:27.670449 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:27.670486 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:39:27.670572 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:39:27.670899 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:39:27.670909 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:39:27.670916 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 18:39:27.671196 (+ 280us) client_negotiation.cc:240] Negotiation successful
0620 18:39:27.671451 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":289,"threads_started":1}
I20260620 18:39:27.673805 1565 heartbeater.cc:347] Connected to a master server at 127.23.115.62:35451
I20260620 18:39:27.674069 1565 heartbeater.cc:464] Registering TS with master...
I20260620 18:39:27.674643 1565 heartbeater.cc:511] Master 127.23.115.62:35451 requested a full tablet report, sending...
I20260620 18:39:27.676918 1379 ts_manager.cc:194] Registered new tserver with Master: 2dd940aad1a541cab7d99929d528d22e (127.23.115.1:33833)
W20260620 18:39:27.986171 1569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:39:27.986519 1569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:39:27.986620 1569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:39:27.993903 1569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:39:27.994004 1569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:39:27.994145 1569 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.115.2
I20260620 18:39:28.004016 1569 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:44057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980676253068-24012-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980676253068-24012-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.115.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980676253068-24012-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.115.2
--webserver_port=0
--tserver_master_addrs=127.23.115.62:35451
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.115.2
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980676253068-24012-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:39:28.006338 1569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:39:28.008450 1569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:39:28.023186 1574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:28.023111 1575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:28.023139 1577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:28.024704 1569 server_base.cc:1034] running on GCE node
I20260620 18:39:28.025381 1569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:39:28.026438 1569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:39:28.027665 1569 hybrid_clock.cc:648] HybridClock initialized: now 1781980768027614 us; error 44 us; skew 500 ppm
I20260620 18:39:28.027987 1569 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:39:28.030158 1569 webserver.cc:460] Webserver started at http://127.23.115.2:41075/ using document root <none> and password file <none>
I20260620 18:39:28.030828 1569 fs_manager.cc:359] Metadata directory not provided
I20260620 18:39:28.030917 1569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:39:28.031244 1569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:39:28.034161 1569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980676253068-24012-0/minicluster-data/ts-1/data/instance:
uuid: "7796b1f700684930bcb5d87d71a71991"
format_stamp: "Formatted at 2026-06-20 18:39:28 on dist-test-slave-d645"
server_key: "f3f8112cc0b02eead662f8701c688a78"
server_key_iv: "945fb3fc342d37cd653c44c6b2e24475"
server_key_version: "encryptionkey@0"
I20260620 18:39:28.034955 1569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980676253068-24012-0/minicluster-data/ts-1/wal/instance:
uuid: "7796b1f700684930bcb5d87d71a71991"
format_stamp: "Formatted at 2026-06-20 18:39:28 on dist-test-slave-d645"
server_key: "f3f8112cc0b02eead662f8701c688a78"
server_key_iv: "945fb3fc342d37cd653c44c6b2e24475"
server_key_version: "encryptionkey@0"
I20260620 18:39:28.040050 1569 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 18:39:28.043603 1583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:28.044950 1569 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:39:28.045115 1569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980676253068-24012-0/minicluster-data/ts-1/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980676253068-24012-0/minicluster-data/ts-1/wal
uuid: "7796b1f700684930bcb5d87d71a71991"
format_stamp: "Formatted at 2026-06-20 18:39:28 on dist-test-slave-d645"
server_key: "f3f8112cc0b02eead662f8701c688a78"
server_key_iv: "945fb3fc342d37cd653c44c6b2e24475"
server_key_version: "encryptionkey@0"
I20260620 18:39:28.045321 1569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980676253068-24012-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980676253068-24012-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980676253068-24012-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:39:28.050781 1569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:39:28.051590 1569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:39:28.051839 1569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:39:28.052747 1569 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:39:28.054116 1569 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:39:28.054208 1569 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:28.054314 1569 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:39:28.054384 1569 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:28.085282 1569 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.2:45049
I20260620 18:39:28.085319 1695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.2:45049 every 8 connection(s)
I20260620 18:39:28.086519 1569 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980676253068-24012-0/minicluster-data/ts-1/data/info.pb
I20260620 18:39:28.088516 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 1569
I20260620 18:39:28.088675 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980676253068-24012-0/minicluster-data/ts-1/wal/instance
I20260620 18:39:28.089077 24012 external_mini_cluster.cc:1428] Setting key d9d23b06ea9a04c0fc48d25a3642a052
I20260620 18:39:28.091820 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980676253068-24012-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980676253068-24012-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980676253068-24012-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980676253068-24012-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.115.3:0
--local_ip_for_outbound_sockets=127.23.115.3
--webserver_interface=127.23.115.3
--webserver_port=0
--tserver_master_addrs=127.23.115.62:35451
--builtin_ntp_servers=127.23.115.20:44057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260620 18:39:28.095095 1567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:28.089504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.2:37101 (local address 127.23.115.62:35451)
0620 18:39:28.089742 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:28.089749 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:28.091321 (+ 1572us) server_negotiation.cc:407] Connection header received
0620 18:39:28.092997 (+ 1676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:28.093004 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:28.093063 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:28.093160 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:28.094434 (+ 1274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:28.094439 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:28.094444 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:39:28.094462 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:28.094497 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:39:28.094511 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:28.094621 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:28.094755 (+ 134us) server_negotiation.cc:299] Negotiation successful
0620 18:39:28.094814 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 18:39:28.095983 1698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:28.090098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:35451 (local address 127.23.115.2:37101)
0620 18:39:28.091126 (+ 1028us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:28.091180 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:28.092712 (+ 1532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:28.093390 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:28.093399 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:28.093496 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:28.094106 (+ 610us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:39:28.094108 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:28.094146 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:39:28.094239 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:39:28.094596 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:39:28.094601 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:39:28.094606 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 18:39:28.094807 (+ 201us) client_negotiation.cc:240] Negotiation successful
0620 18:39:28.095044 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":308,"threads_started":1}
I20260620 18:39:28.097282 1696 heartbeater.cc:347] Connected to a master server at 127.23.115.62:35451
I20260620 18:39:28.097553 1696 heartbeater.cc:464] Registering TS with master...
I20260620 18:39:28.098119 1696 heartbeater.cc:511] Master 127.23.115.62:35451 requested a full tablet report, sending...
I20260620 18:39:28.099570 1379 ts_manager.cc:194] Registered new tserver with Master: 7796b1f700684930bcb5d87d71a71991 (127.23.115.2:45049)
W20260620 18:39:28.413937 1699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:39:28.414223 1699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:39:28.414409 1699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:39:28.421883 1699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:39:28.421985 1699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:39:28.422122 1699 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.115.3
I20260620 18:39:28.431805 1699 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:44057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980676253068-24012-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980676253068-24012-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.115.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980676253068-24012-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.115.3
--webserver_port=0
--tserver_master_addrs=127.23.115.62:35451
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.115.3
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980676253068-24012-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:39:28.433064 1699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:39:28.434814 1699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:39:28.447599 1705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:28.447643 1704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:28.447556 1707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:28.449030 1699 server_base.cc:1034] running on GCE node
I20260620 18:39:28.449702 1699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:39:28.450796 1699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:39:28.452041 1699 hybrid_clock.cc:648] HybridClock initialized: now 1781980768451973 us; error 54 us; skew 500 ppm
I20260620 18:39:28.452382 1699 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:39:28.454753 1699 webserver.cc:460] Webserver started at http://127.23.115.3:37093/ using document root <none> and password file <none>
I20260620 18:39:28.455471 1699 fs_manager.cc:359] Metadata directory not provided
I20260620 18:39:28.455571 1699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:39:28.455883 1699 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:39:28.458722 1699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980676253068-24012-0/minicluster-data/ts-2/data/instance:
uuid: "8f7408874cff496f89f11636bb289999"
format_stamp: "Formatted at 2026-06-20 18:39:28 on dist-test-slave-d645"
server_key: "9a0650121b578151e7464d3a4189f069"
server_key_iv: "e84acd0f900c43b0105c7716b2ad8acd"
server_key_version: "encryptionkey@0"
I20260620 18:39:28.459609 1699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980676253068-24012-0/minicluster-data/ts-2/wal/instance:
uuid: "8f7408874cff496f89f11636bb289999"
format_stamp: "Formatted at 2026-06-20 18:39:28 on dist-test-slave-d645"
server_key: "9a0650121b578151e7464d3a4189f069"
server_key_iv: "e84acd0f900c43b0105c7716b2ad8acd"
server_key_version: "encryptionkey@0"
I20260620 18:39:28.465611 1699 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260620 18:39:28.470067 1713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:28.471906 1699 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 18:39:28.472078 1699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980676253068-24012-0/minicluster-data/ts-2/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980676253068-24012-0/minicluster-data/ts-2/wal
uuid: "8f7408874cff496f89f11636bb289999"
format_stamp: "Formatted at 2026-06-20 18:39:28 on dist-test-slave-d645"
server_key: "9a0650121b578151e7464d3a4189f069"
server_key_iv: "e84acd0f900c43b0105c7716b2ad8acd"
server_key_version: "encryptionkey@0"
I20260620 18:39:28.472340 1699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980676253068-24012-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980676253068-24012-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980676253068-24012-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:39:28.478123 1699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:39:28.478912 1699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:39:28.479202 1699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:39:28.480230 1699 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:39:28.481674 1699 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:39:28.481768 1699 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:28.481874 1699 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:39:28.481941 1699 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:28.513835 1699 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.3:42101
I20260620 18:39:28.513902 1825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.3:42101 every 8 connection(s)
I20260620 18:39:28.515120 1699 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980676253068-24012-0/minicluster-data/ts-2/data/info.pb
I20260620 18:39:28.523382 1567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:28.517619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.3:33313 (local address 127.23.115.62:35451)
0620 18:39:28.517847 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:28.517853 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:28.519186 (+ 1333us) server_negotiation.cc:407] Connection header received
0620 18:39:28.520800 (+ 1614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:28.520805 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:28.520864 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:28.520958 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:28.522313 (+ 1355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:28.522319 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:28.522323 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:39:28.522340 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:28.522371 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:39:28.522384 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:28.522484 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:28.522817 (+ 333us) server_negotiation.cc:299] Negotiation successful
0620 18:39:28.523025 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260620 18:39:28.523670 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 1699
I20260620 18:39:28.523860 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980676253068-24012-0/minicluster-data/ts-2/wal/instance
I20260620 18:39:28.524123 1828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:28.518096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:35451 (local address 127.23.115.3:33313)
0620 18:39:28.519029 (+ 933us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:28.519080 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:28.520540 (+ 1460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:28.521193 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:28.521204 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:28.521297 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:28.522007 (+ 710us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:39:28.522009 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:28.522046 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:39:28.522128 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:39:28.522480 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:39:28.522488 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:39:28.522493 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 18:39:28.522702 (+ 209us) client_negotiation.cc:240] Negotiation successful
0620 18:39:28.522907 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":278,"threads_started":1}
I20260620 18:39:28.524399 24012 external_mini_cluster.cc:1428] Setting key b02c7a38317dab7bcd6c67106ba3da43
I20260620 18:39:28.525499 1826 heartbeater.cc:347] Connected to a master server at 127.23.115.62:35451
I20260620 18:39:28.525849 1826 heartbeater.cc:464] Registering TS with master...
I20260620 18:39:28.526500 1826 heartbeater.cc:511] Master 127.23.115.62:35451 requested a full tablet report, sending...
I20260620 18:39:28.528163 1379 ts_manager.cc:194] Registered new tserver with Master: 8f7408874cff496f89f11636bb289999 (127.23.115.3:42101)
I20260620 18:39:28.540457 24012 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 18:39:28.547097 1567 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:39:28.545102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56330 (local address 127.23.115.62:35451)
0620 18:39:28.545252 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:28.545257 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:28.545697 (+ 440us) server_negotiation.cc:407] Connection header received
0620 18:39:28.545982 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:28.545988 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:28.546079 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:28.546214 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:28.546615 (+ 401us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56330: BlockingRecv error: recv got EOF from 127.0.0.1:56330 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 18:39:28.547760 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 1438
I20260620 18:39:28.551460 1835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:28.548722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54840 (local address 127.23.115.1:33833)
0620 18:39:28.549185 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:28.549193 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:28.549215 (+ 22us) server_negotiation.cc:407] Connection header received
0620 18:39:28.549279 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:28.549285 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:28.549527 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:28.549687 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:28.550687 (+ 1000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:28.550696 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:28.550710 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:39:28.550730 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:28.550770 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:39:28.550784 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:28.550898 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:28.551039 (+ 141us) server_negotiation.cc:299] Negotiation successful
0620 18:39:28.551135 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":202,"threads_started":1}
I20260620 18:39:28.552660 1560 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:39:28.633104 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 1438
I20260620 18:39:28.647019 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 1569
I20260620 18:39:28.650630 1837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:28.648008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41078 (local address 127.23.115.2:45049)
0620 18:39:28.648567 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:28.648576 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:28.648602 (+ 26us) server_negotiation.cc:407] Connection header received
0620 18:39:28.648665 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:28.648671 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:28.648904 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:28.649055 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:28.649899 (+ 844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:28.649905 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:28.649916 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:39:28.649932 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:28.649969 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:39:28.649982 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:28.650108 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:28.650269 (+ 161us) server_negotiation.cc:299] Negotiation successful
0620 18:39:28.650367 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":230,"threads_started":1}
I20260620 18:39:28.651988 1691 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:39:28.737020 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 1569
I20260620 18:39:28.750653 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 1699
I20260620 18:39:28.754451 1839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:28.751642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36356 (local address 127.23.115.3:42101)
0620 18:39:28.752227 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:28.752234 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:28.752274 (+ 40us) server_negotiation.cc:407] Connection header received
0620 18:39:28.752335 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:28.752341 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:28.752544 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:28.752669 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:28.753486 (+ 817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:28.753492 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:28.753502 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:39:28.753513 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:28.753553 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:39:28.753568 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:28.753698 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:28.754025 (+ 327us) server_negotiation.cc:299] Negotiation successful
0620 18:39:28.754120 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":291,"threads_started":1}
I20260620 18:39:28.755651 1821 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:39:28.836041 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 1699
I20260620 18:39:28.849049 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 1348
I20260620 18:39:28.850351 1409 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:39:28.950680 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 1348
2026-06-20T18:39:28Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2327 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980676253068-24012-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:39:29 dist-test-slave-d645 krb5kdc[1844](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:39:29 dist-test-slave-d645 krb5kdc[1844](info): set up 2 sockets
Jun 20 18:39:29 dist-test-slave-d645 krb5kdc[1844](info): commencing operation
krb5kdc: starting...
W20260620 18:39:29.972275 24012 mini_kdc.cc:121] Time spent starting KDC: real 0.984s user 0.002s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980676253068-24012-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980676253068-24012-0/krb5kdc/test-user.keytab.
Jun 20 18:39:30 dist-test-slave-d645 krb5kdc[1844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980770, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:39:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:39:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.115.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.115.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.115.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab.
Entry for principal kudu/127.23.115.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab.
I20260620 18:39:30.198199 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980676253068-24012-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980676253068-24012-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980676253068-24012-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.115.62:45983
--webserver_interface=127.23.115.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab
--principal=kudu/127.23.115.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:35725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.115.62:45983
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:39:30.519411 1857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:39:30.519743 1857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:39:30.519881 1857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:39:30.527365 1857 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:39:30.527504 1857 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:39:30.527590 1857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:39:30.527667 1857 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:39:30.527733 1857 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:39:30.537892 1857 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:35725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980676253068-24012-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980676253068-24012-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.115.62:45983
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab
--principal=kudu/127.23.115.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.115.62:45983
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.115.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980676253068-24012-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:39:30.539273 1857 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:39:30.540594 1857 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:39:30.552634 1862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:30.552802 1865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:30.552806 1863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:30.553292 1857 server_base.cc:1034] running on GCE node
I20260620 18:39:30.554093 1857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:39:30.555583 1857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:39:30.556828 1857 hybrid_clock.cc:648] HybridClock initialized: now 1781980770556762 us; error 50 us; skew 500 ppm
I20260620 18:39:30.557196 1857 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:39:30 dist-test-slave-d645 krb5kdc[1844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980770, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:39:30.566377 1857 init.cc:377] Logged in from keytab as kudu/127.23.115.62@KRBTEST.COM (short username kudu)
I20260620 18:39:30.567988 1857 webserver.cc:460] Webserver started at http://127.23.115.62:46503/ using document root <none> and password file <none>
I20260620 18:39:30.568718 1857 fs_manager.cc:359] Metadata directory not provided
I20260620 18:39:30.568827 1857 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:39:30.569083 1857 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:39:30.572043 1857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980676253068-24012-0/minicluster-data/master-0/data/instance:
uuid: "e9bb4fa99e964acba47fadd442ee8f14"
format_stamp: "Formatted at 2026-06-20 18:39:30 on dist-test-slave-d645"
server_key: "1870c92a02b2fa46803ff4eb2ab1616b"
server_key_iv: "b6f320bc4cbe07a994439e98e6423753"
server_key_version: "encryptionkey@0"
I20260620 18:39:30.572772 1857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980676253068-24012-0/minicluster-data/master-0/wal/instance:
uuid: "e9bb4fa99e964acba47fadd442ee8f14"
format_stamp: "Formatted at 2026-06-20 18:39:30 on dist-test-slave-d645"
server_key: "1870c92a02b2fa46803ff4eb2ab1616b"
server_key_iv: "b6f320bc4cbe07a994439e98e6423753"
server_key_version: "encryptionkey@0"
I20260620 18:39:30.578070 1857 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 18:39:30.581912 1872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:30.583447 1857 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 18:39:30.583600 1857 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980676253068-24012-0/minicluster-data/master-0/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980676253068-24012-0/minicluster-data/master-0/wal
uuid: "e9bb4fa99e964acba47fadd442ee8f14"
format_stamp: "Formatted at 2026-06-20 18:39:30 on dist-test-slave-d645"
server_key: "1870c92a02b2fa46803ff4eb2ab1616b"
server_key_iv: "b6f320bc4cbe07a994439e98e6423753"
server_key_version: "encryptionkey@0"
I20260620 18:39:30.583770 1857 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980676253068-24012-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980676253068-24012-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980676253068-24012-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:39:30.613049 1857 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:39:30.617264 1857 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:39:30.617511 1857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:39:30.635216 1857 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.62:45983
I20260620 18:39:30.635265 1923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.62:45983 every 8 connection(s)
I20260620 18:39:30.636644 1857 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
I20260620 18:39:30.639981 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 1857
I20260620 18:39:30.640196 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980676253068-24012-0/minicluster-data/master-0/wal/instance
I20260620 18:39:30.640565 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:39:30.640837 24012 external_mini_cluster.cc:1428] Setting key 325ae3002898d06caa15dec1009b4b41
I20260620 18:39:30.653458 1924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9bb4fa99e964acba47fadd442ee8f14: Bootstrap starting.
Jun 20 18:39:30 dist-test-slave-d645 krb5kdc[1844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980770, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.115.62@KRBTEST.COM
I20260620 18:39:30.657132 1924 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e9bb4fa99e964acba47fadd442ee8f14: Neither blocks nor log segments found. Creating new log.
I20260620 18:39:30.658291 1924 log.cc:824] T 00000000000000000000000000000000 P e9bb4fa99e964acba47fadd442ee8f14: Log is configured to *not* fsync() on all Append() calls
I20260620 18:39:30.661129 1924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9bb4fa99e964acba47fadd442ee8f14: No bootstrap required, opened a new log
I20260620 18:39:30.663489 1927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:30.642819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54890 (local address 127.23.115.62:45983)
0620 18:39:30.643639 (+ 820us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:30.643651 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:30.643695 (+ 44us) server_negotiation.cc:407] Connection header received
0620 18:39:30.645696 (+ 2001us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:30.645719 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:30.646120 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:30.646499 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:30.648209 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.649487 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.650739 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:30.651501 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:30.651668 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:30.656316 (+ 4648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:30.656325 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:30.656340 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:30.656380 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:30.659651 (+ 3271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:30.660302 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:30.660308 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:30.660315 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:30.660471 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:30.660817 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:30.660821 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:30.660823 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:30.661367 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:30.661664 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:30.662050 (+ 386us) server_negotiation.cc:299] Negotiation successful
0620 18:39:30.662462 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":338,"threads_started":1}
I20260620 18:39:30.669458 1924 raft_consensus.cc:348] T 00000000000000000000000000000000 P e9bb4fa99e964acba47fadd442ee8f14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9bb4fa99e964acba47fadd442ee8f14" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 45983 } }
I20260620 18:39:30.669792 1924 raft_consensus.cc:374] T 00000000000000000000000000000000 P e9bb4fa99e964acba47fadd442ee8f14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:39:30.669893 1924 raft_consensus.cc:729] T 00000000000000000000000000000000 P e9bb4fa99e964acba47fadd442ee8f14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9bb4fa99e964acba47fadd442ee8f14, State: Initialized, Role: FOLLOWER
I20260620 18:39:30.670461 1924 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9bb4fa99e964acba47fadd442ee8f14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9bb4fa99e964acba47fadd442ee8f14" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 45983 } }
I20260620 18:39:30.670670 1924 raft_consensus.cc:388] T 00000000000000000000000000000000 P e9bb4fa99e964acba47fadd442ee8f14 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:39:30.670792 1924 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9bb4fa99e964acba47fadd442ee8f14 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:39:30.670970 1924 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e9bb4fa99e964acba47fadd442ee8f14 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:39:30.673156 1924 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9bb4fa99e964acba47fadd442ee8f14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9bb4fa99e964acba47fadd442ee8f14" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 45983 } }
I20260620 18:39:30.673703 1924 leader_election.cc:302] T 00000000000000000000000000000000 P e9bb4fa99e964acba47fadd442ee8f14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e9bb4fa99e964acba47fadd442ee8f14; no voters:
I20260620 18:39:30.674337 1924 leader_election.cc:288] T 00000000000000000000000000000000 P e9bb4fa99e964acba47fadd442ee8f14 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:39:30.674467 1929 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e9bb4fa99e964acba47fadd442ee8f14 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:39:30.675228 1929 raft_consensus.cc:686] T 00000000000000000000000000000000 P e9bb4fa99e964acba47fadd442ee8f14 [term 1 LEADER]: Becoming Leader. State: Replica: e9bb4fa99e964acba47fadd442ee8f14, State: Running, Role: LEADER
I20260620 18:39:30.675704 1929 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9bb4fa99e964acba47fadd442ee8f14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9bb4fa99e964acba47fadd442ee8f14" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 45983 } }
I20260620 18:39:30.676350 1924 sys_catalog.cc:564] T 00000000000000000000000000000000 P e9bb4fa99e964acba47fadd442ee8f14 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:39:30.678829 1931 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9bb4fa99e964acba47fadd442ee8f14 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9bb4fa99e964acba47fadd442ee8f14. Latest consensus state: current_term: 1 leader_uuid: "e9bb4fa99e964acba47fadd442ee8f14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9bb4fa99e964acba47fadd442ee8f14" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 45983 } } }
I20260620 18:39:30.679267 1931 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9bb4fa99e964acba47fadd442ee8f14 [sys.catalog]: This master's current role is: LEADER
I20260620 18:39:30.679064 1930 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9bb4fa99e964acba47fadd442ee8f14 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e9bb4fa99e964acba47fadd442ee8f14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9bb4fa99e964acba47fadd442ee8f14" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 45983 } } }
I20260620 18:39:30.679363 1930 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9bb4fa99e964acba47fadd442ee8f14 [sys.catalog]: This master's current role is: LEADER
I20260620 18:39:30.680208 1937 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:39:30.685822 1937 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:39:30.694463 1937 catalog_manager.cc:1347] Generated new cluster ID: e33d5fb60bff4ffeadf452a0fd0e4f7c
I20260620 18:39:30.694571 1937 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:39:30.743647 1937 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:39:30.745497 1937 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:39:30.754783 1937 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e9bb4fa99e964acba47fadd442ee8f14: Generated new TSK 0
I20260620 18:39:30.755957 1937 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.115.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.115.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.115.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.1.keytab.
Entry for principal kudu/127.23.115.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.1.keytab.
I20260620 18:39:30.819943 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980676253068-24012-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980676253068-24012-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980676253068-24012-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980676253068-24012-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.115.1:0
--local_ip_for_outbound_sockets=127.23.115.1
--webserver_interface=127.23.115.1
--webserver_port=0
--tserver_master_addrs=127.23.115.62:45983
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.1.keytab
--principal=kudu/127.23.115.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:35725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:39:31.161010 1950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:39:31.161294 1950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:39:31.161407 1950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:39:31.169431 1950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:39:31.169554 1950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:39:31.169704 1950 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.115.1
I20260620 18:39:31.180138 1950 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:35725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980676253068-24012-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980676253068-24012-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.1.keytab
--principal=kudu/127.23.115.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.115.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980676253068-24012-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.115.1
--webserver_port=0
--tserver_master_addrs=127.23.115.62:45983
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.115.1
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980676253068-24012-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:39:31.181479 1950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:39:31.182811 1950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:39:31.195391 1955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:31.195765 1956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:31.195937 1958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:31.196887 1950 server_base.cc:1034] running on GCE node
I20260620 18:39:31.197458 1950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:39:31.198333 1950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:39:31.199592 1950 hybrid_clock.cc:648] HybridClock initialized: now 1781980771199505 us; error 48 us; skew 500 ppm
I20260620 18:39:31.199894 1950 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:39:31 dist-test-slave-d645 krb5kdc[1844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980771, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:39:31.203675 1950 init.cc:377] Logged in from keytab as kudu/127.23.115.1@KRBTEST.COM (short username kudu)
I20260620 18:39:31.205189 1950 webserver.cc:460] Webserver started at http://127.23.115.1:34067/ using document root <none> and password file <none>
I20260620 18:39:31.205870 1950 fs_manager.cc:359] Metadata directory not provided
I20260620 18:39:31.205973 1950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:39:31.206261 1950 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:39:31.209197 1950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980676253068-24012-0/minicluster-data/ts-0/data/instance:
uuid: "6103605c1ae34c638dc21ba5a8b8e0a9"
format_stamp: "Formatted at 2026-06-20 18:39:31 on dist-test-slave-d645"
server_key: "73ca0bcd8912c2e727ad17220c7d7fad"
server_key_iv: "a6ee05abae7104441e94f9fd61bccff7"
server_key_version: "encryptionkey@0"
I20260620 18:39:31.209928 1950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980676253068-24012-0/minicluster-data/ts-0/wal/instance:
uuid: "6103605c1ae34c638dc21ba5a8b8e0a9"
format_stamp: "Formatted at 2026-06-20 18:39:31 on dist-test-slave-d645"
server_key: "73ca0bcd8912c2e727ad17220c7d7fad"
server_key_iv: "a6ee05abae7104441e94f9fd61bccff7"
server_key_version: "encryptionkey@0"
I20260620 18:39:31.215138 1950 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260620 18:39:31.218560 1965 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:31.220151 1950 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:39:31.220320 1950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980676253068-24012-0/minicluster-data/ts-0/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980676253068-24012-0/minicluster-data/ts-0/wal
uuid: "6103605c1ae34c638dc21ba5a8b8e0a9"
format_stamp: "Formatted at 2026-06-20 18:39:31 on dist-test-slave-d645"
server_key: "73ca0bcd8912c2e727ad17220c7d7fad"
server_key_iv: "a6ee05abae7104441e94f9fd61bccff7"
server_key_version: "encryptionkey@0"
I20260620 18:39:31.220605 1950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980676253068-24012-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980676253068-24012-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980676253068-24012-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:39:31.242523 1950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:39:31.246579 1950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:39:31.247001 1950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:39:31.248008 1950 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:39:31.249573 1950 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:39:31.249655 1950 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:31.249780 1950 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:39:31.249845 1950 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:31.284006 1950 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.1:42777
I20260620 18:39:31.284080 2077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.1:42777 every 8 connection(s)
I20260620 18:39:31.285272 1950 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980676253068-24012-0/minicluster-data/ts-0/data/info.pb
I20260620 18:39:31.293213 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 1950
I20260620 18:39:31.293426 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980676253068-24012-0/minicluster-data/ts-0/wal/instance
I20260620 18:39:31.294082 24012 external_mini_cluster.cc:1428] Setting key 59e021e7a338e8cd0d873d0826575587
Jun 20 18:39:31 dist-test-slave-d645 krb5kdc[1844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980771, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.1@KRBTEST.COM for kudu/127.23.115.62@KRBTEST.COM
I20260620 18:39:31.306854 2080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:31.287865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.1:54345 (local address 127.23.115.62:45983)
0620 18:39:31.288351 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:31.288357 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:31.289554 (+ 1197us) server_negotiation.cc:407] Connection header received
0620 18:39:31.291481 (+ 1927us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:31.291487 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:31.291575 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:31.291671 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:31.293180 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:31.293986 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:31.295700 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:31.296122 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:31.296253 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:31.299881 (+ 3628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:31.299887 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:31.299890 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:31.299924 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:31.302436 (+ 2512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:31.303304 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:31.303309 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:31.303311 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:31.303540 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:31.304185 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:31.304192 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:31.304195 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:31.304736 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:31.304963 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:31.305995 (+ 1032us) server_negotiation.cc:299] Negotiation successful
0620 18:39:31.306474 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":220,"threads_started":1}
I20260620 18:39:31.307919 2081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:31.288219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:45983 (local address 127.23.115.1:54345)
0620 18:39:31.289231 (+ 1012us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:31.289311 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:31.291220 (+ 1909us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:31.291917 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:31.291924 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:31.292433 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:31.293012 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:31.293033 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:31.294144 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:31.294148 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:31.295555 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:31.295568 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:31.296343 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:31.296350 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:31.296574 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:31.297245 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:31.297268 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:31.299617 (+ 2349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:39:31.302589 (+ 2972us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:31.302598 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:31.302611 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:31.303144 (+ 533us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:31.303675 (+ 531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:31.303678 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:31.303681 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:31.303888 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:31.304986 (+ 1098us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:39:31.304998 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:39:31.305519 (+ 521us) client_negotiation.cc:769] Sending connection context
0620 18:39:31.305832 (+ 313us) client_negotiation.cc:240] Negotiation successful
0620 18:39:31.306638 (+ 806us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":669,"thread_start_us":392,"threads_started":1}
I20260620 18:39:31.309609 2078 heartbeater.cc:347] Connected to a master server at 127.23.115.62:45983
I20260620 18:39:31.309899 2078 heartbeater.cc:464] Registering TS with master...
I20260620 18:39:31.310627 2078 heartbeater.cc:511] Master 127.23.115.62:45983 requested a full tablet report, sending...
I20260620 18:39:31.312958 1889 ts_manager.cc:194] Registered new tserver with Master: 6103605c1ae34c638dc21ba5a8b8e0a9 (127.23.115.1:42777)
WARNING: no policy specified for kudu/127.23.115.2@KRBTEST.COM; defaulting to no policy
I20260620 18:39:31.314786 1889 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.115.1@KRBTEST.COM'} at 127.23.115.1:54345
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.115.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.115.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.2.keytab.
Entry for principal kudu/127.23.115.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.2.keytab.
I20260620 18:39:31.343871 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980676253068-24012-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980676253068-24012-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980676253068-24012-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980676253068-24012-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.115.2:0
--local_ip_for_outbound_sockets=127.23.115.2
--webserver_interface=127.23.115.2
--webserver_port=0
--tserver_master_addrs=127.23.115.62:45983
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.2.keytab
--principal=kudu/127.23.115.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:35725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:39:31.663275 2084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:39:31.663570 2084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:39:31.663700 2084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:39:31.670955 2084 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:39:31.671075 2084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:39:31.671223 2084 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.115.2
I20260620 18:39:31.682446 2084 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:35725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980676253068-24012-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980676253068-24012-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.2.keytab
--principal=kudu/127.23.115.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.115.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980676253068-24012-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.115.2
--webserver_port=0
--tserver_master_addrs=127.23.115.62:45983
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.115.2
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980676253068-24012-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:39:31.683811 2084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:39:31.685070 2084 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:39:31.698282 2089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:31.698303 2092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:31.698354 2090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:31.699062 2084 server_base.cc:1034] running on GCE node
I20260620 18:39:31.699712 2084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:39:31.700666 2084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:39:31.701874 2084 hybrid_clock.cc:648] HybridClock initialized: now 1781980771701811 us; error 52 us; skew 500 ppm
I20260620 18:39:31.702205 2084 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:39:31 dist-test-slave-d645 krb5kdc[1844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980771, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:39:31.705641 2084 init.cc:377] Logged in from keytab as kudu/127.23.115.2@KRBTEST.COM (short username kudu)
I20260620 18:39:31.707106 2084 webserver.cc:460] Webserver started at http://127.23.115.2:37005/ using document root <none> and password file <none>
I20260620 18:39:31.707782 2084 fs_manager.cc:359] Metadata directory not provided
I20260620 18:39:31.707880 2084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:39:31.708158 2084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:39:31.711015 2084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980676253068-24012-0/minicluster-data/ts-1/data/instance:
uuid: "4c86ce0e45f64483bcf286c12b1bbd2f"
format_stamp: "Formatted at 2026-06-20 18:39:31 on dist-test-slave-d645"
server_key: "5ac0bf30bc5f1e540d1eb06670e4f3e0"
server_key_iv: "8361607ae7cafde7c2ed73bac2dd43e8"
server_key_version: "encryptionkey@0"
I20260620 18:39:31.711750 2084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980676253068-24012-0/minicluster-data/ts-1/wal/instance:
uuid: "4c86ce0e45f64483bcf286c12b1bbd2f"
format_stamp: "Formatted at 2026-06-20 18:39:31 on dist-test-slave-d645"
server_key: "5ac0bf30bc5f1e540d1eb06670e4f3e0"
server_key_iv: "8361607ae7cafde7c2ed73bac2dd43e8"
server_key_version: "encryptionkey@0"
I20260620 18:39:31.716862 2084 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:39:31.720012 2099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:31.721339 2084 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:39:31.721488 2084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980676253068-24012-0/minicluster-data/ts-1/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980676253068-24012-0/minicluster-data/ts-1/wal
uuid: "4c86ce0e45f64483bcf286c12b1bbd2f"
format_stamp: "Formatted at 2026-06-20 18:39:31 on dist-test-slave-d645"
server_key: "5ac0bf30bc5f1e540d1eb06670e4f3e0"
server_key_iv: "8361607ae7cafde7c2ed73bac2dd43e8"
server_key_version: "encryptionkey@0"
I20260620 18:39:31.721657 2084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980676253068-24012-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980676253068-24012-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980676253068-24012-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:39:31.743573 2084 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:39:31.747503 2084 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:39:31.747778 2084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:39:31.748675 2084 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:39:31.749979 2084 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:39:31.750057 2084 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:31.750173 2084 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:39:31.750224 2084 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:31.782678 2084 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.2:42135
I20260620 18:39:31.782743 2211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.2:42135 every 8 connection(s)
I20260620 18:39:31.783949 2084 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980676253068-24012-0/minicluster-data/ts-1/data/info.pb
I20260620 18:39:31.785892 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 2084
I20260620 18:39:31.786059 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980676253068-24012-0/minicluster-data/ts-1/wal/instance
I20260620 18:39:31.786454 24012 external_mini_cluster.cc:1428] Setting key 70ea951a9675347e27349a4c5aced9ca
Jun 20 18:39:31 dist-test-slave-d645 krb5kdc[1844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980771, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.2@KRBTEST.COM for kudu/127.23.115.62@KRBTEST.COM
I20260620 18:39:31.805495 2080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:31.787203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.2:53015 (local address 127.23.115.62:45983)
0620 18:39:31.787368 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:31.787375 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:31.787994 (+ 619us) server_negotiation.cc:407] Connection header received
0620 18:39:31.789895 (+ 1901us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:31.789902 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:31.789985 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:31.790098 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:31.791962 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:31.792940 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:31.795612 (+ 2672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:31.796094 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:31.796211 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:31.799997 (+ 3786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:31.800003 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:31.800006 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:31.800037 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:31.802421 (+ 2384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:31.803168 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:31.803172 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:31.803174 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:31.803291 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:31.803712 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:31.803716 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:31.803720 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:31.803988 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:31.804179 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:31.804942 (+ 763us) server_negotiation.cc:299] Negotiation successful
0620 18:39:31.805168 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 18:39:31.806484 2214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:31.786991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:45983 (local address 127.23.115.2:53015)
0620 18:39:31.787847 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:31.787891 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:31.789632 (+ 1741us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:31.790374 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:31.790383 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:31.791136 (+ 753us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:31.791737 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:31.791756 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:31.793222 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:31.793230 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:31.795435 (+ 2205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:31.795456 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:31.796331 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:31.796340 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:31.796576 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:31.797217 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:31.797246 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:31.799759 (+ 2513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:39:31.802570 (+ 2811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:31.802580 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:31.802597 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:31.803053 (+ 456us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:31.803406 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:31.803411 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:31.803415 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:31.803590 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:31.804108 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:39:31.804115 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:39:31.804546 (+ 431us) client_negotiation.cc:769] Sending connection context
0620 18:39:31.804808 (+ 262us) client_negotiation.cc:240] Negotiation successful
0620 18:39:31.805356 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":330,"threads_started":1}
I20260620 18:39:31.807991 2212 heartbeater.cc:347] Connected to a master server at 127.23.115.62:45983
I20260620 18:39:31.808266 2212 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.23.115.3@KRBTEST.COM; defaulting to no policy
I20260620 18:39:31.809002 2212 heartbeater.cc:511] Master 127.23.115.62:45983 requested a full tablet report, sending...
I20260620 18:39:31.810401 1889 ts_manager.cc:194] Registered new tserver with Master: 4c86ce0e45f64483bcf286c12b1bbd2f (127.23.115.2:42135)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.115.3@KRBTEST.COM" created.
I20260620 18:39:31.811394 1889 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.115.2@KRBTEST.COM'} at 127.23.115.2:53015
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.115.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.3.keytab.
Entry for principal kudu/127.23.115.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.3.keytab.
I20260620 18:39:31.837872 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980676253068-24012-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980676253068-24012-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980676253068-24012-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980676253068-24012-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.115.3:0
--local_ip_for_outbound_sockets=127.23.115.3
--webserver_interface=127.23.115.3
--webserver_port=0
--tserver_master_addrs=127.23.115.62:45983
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.3.keytab
--principal=kudu/127.23.115.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:35725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:39:32.152236 2217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:39:32.152503 2217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:39:32.152622 2217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:39:32.159682 2217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:39:32.159793 2217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:39:32.159938 2217 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.115.3
I20260620 18:39:32.169517 2217 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:35725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980676253068-24012-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980676253068-24012-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.3.keytab
--principal=kudu/127.23.115.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.115.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980676253068-24012-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.115.3
--webserver_port=0
--tserver_master_addrs=127.23.115.62:45983
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.115.3
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980676253068-24012-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:39:32.170769 2217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:39:32.172025 2217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:39:32.184381 2222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:32.184479 2225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:39:32.184561 2223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:39:32.184981 2217 server_base.cc:1034] running on GCE node
I20260620 18:39:32.185572 2217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:39:32.186580 2217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:39:32.187891 2217 hybrid_clock.cc:648] HybridClock initialized: now 1781980772187844 us; error 51 us; skew 500 ppm
I20260620 18:39:32.188244 2217 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:39:32 dist-test-slave-d645 krb5kdc[1844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980772, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:39:32.191371 2217 init.cc:377] Logged in from keytab as kudu/127.23.115.3@KRBTEST.COM (short username kudu)
I20260620 18:39:32.192785 2217 webserver.cc:460] Webserver started at http://127.23.115.3:45321/ using document root <none> and password file <none>
I20260620 18:39:32.193445 2217 fs_manager.cc:359] Metadata directory not provided
I20260620 18:39:32.193557 2217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:39:32.193835 2217 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:39:32.196918 2217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980676253068-24012-0/minicluster-data/ts-2/data/instance:
uuid: "28ed4c6ac97348efbca01275baebff16"
format_stamp: "Formatted at 2026-06-20 18:39:32 on dist-test-slave-d645"
server_key: "1122c7f443470153730a98603a91ffdc"
server_key_iv: "cd890cac790b687e00ff55b8dd0f65a7"
server_key_version: "encryptionkey@0"
I20260620 18:39:32.197686 2217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980676253068-24012-0/minicluster-data/ts-2/wal/instance:
uuid: "28ed4c6ac97348efbca01275baebff16"
format_stamp: "Formatted at 2026-06-20 18:39:32 on dist-test-slave-d645"
server_key: "1122c7f443470153730a98603a91ffdc"
server_key_iv: "cd890cac790b687e00ff55b8dd0f65a7"
server_key_version: "encryptionkey@0"
I20260620 18:39:32.202653 2217 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 18:39:32.206255 2232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:32.207624 2217 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:39:32.207770 2217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980676253068-24012-0/minicluster-data/ts-2/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980676253068-24012-0/minicluster-data/ts-2/wal
uuid: "28ed4c6ac97348efbca01275baebff16"
format_stamp: "Formatted at 2026-06-20 18:39:32 on dist-test-slave-d645"
server_key: "1122c7f443470153730a98603a91ffdc"
server_key_iv: "cd890cac790b687e00ff55b8dd0f65a7"
server_key_version: "encryptionkey@0"
I20260620 18:39:32.207986 2217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980676253068-24012-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980676253068-24012-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980676253068-24012-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:39:32.243801 2217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:39:32.247964 2217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:39:32.248261 2217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:39:32.249213 2217 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:39:32.250618 2217 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:39:32.250696 2217 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:32.250824 2217 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:39:32.250877 2217 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:39:32.284677 2217 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.3:46435
I20260620 18:39:32.284725 2344 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.3:46435 every 8 connection(s)
I20260620 18:39:32.286218 2217 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980676253068-24012-0/minicluster-data/ts-2/data/info.pb
I20260620 18:39:32.290979 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 2217
I20260620 18:39:32.291172 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980676253068-24012-0/minicluster-data/ts-2/wal/instance
I20260620 18:39:32.291566 24012 external_mini_cluster.cc:1428] Setting key 3b08edde696d2b795920b24a10bbd5f6
Jun 20 18:39:32 dist-test-slave-d645 krb5kdc[1844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980772, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.3@KRBTEST.COM for kudu/127.23.115.62@KRBTEST.COM
I20260620 18:39:32.306883 2080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:32.288963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.3:34349 (local address 127.23.115.62:45983)
0620 18:39:32.289157 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:32.289163 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:32.290303 (+ 1140us) server_negotiation.cc:407] Connection header received
0620 18:39:32.292391 (+ 2088us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:32.292395 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:32.292534 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:32.292636 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:32.294439 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:32.295477 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:32.297099 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:32.297531 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:32.297633 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:32.301239 (+ 3606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:32.301245 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:32.301249 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:32.301287 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:32.303812 (+ 2525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:32.304584 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:32.304588 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:32.304590 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:32.304720 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:32.305106 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:32.305109 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:32.305112 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:32.305389 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:32.305496 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:32.306392 (+ 896us) server_negotiation.cc:299] Negotiation successful
0620 18:39:32.306600 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 18:39:32.308192 2347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:32.289384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:45983 (local address 127.23.115.3:34349)
0620 18:39:32.290135 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:32.290178 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:32.291995 (+ 1817us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:32.292974 (+ 979us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:32.292987 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:32.293691 (+ 704us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:39:32.294249 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:32.294267 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:32.295638 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:32.295642 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:32.296947 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:32.296959 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:32.297751 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:32.297756 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:32.297975 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:32.298619 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:39:32.298648 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:39:32.301005 (+ 2357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:39:32.303971 (+ 2966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:32.303978 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:32.303992 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:32.304456 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:32.304831 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:39:32.304834 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:39:32.304836 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:39:32.305022 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:39:32.305521 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:39:32.305529 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:39:32.305957 (+ 428us) client_negotiation.cc:769] Sending connection context
0620 18:39:32.306269 (+ 312us) client_negotiation.cc:240] Negotiation successful
0620 18:39:32.306802 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":317,"threads_started":1}
I20260620 18:39:32.309854 2345 heartbeater.cc:347] Connected to a master server at 127.23.115.62:45983
I20260620 18:39:32.310173 2345 heartbeater.cc:464] Registering TS with master...
I20260620 18:39:32.310987 2345 heartbeater.cc:511] Master 127.23.115.62:45983 requested a full tablet report, sending...
I20260620 18:39:32.312783 1889 ts_manager.cc:194] Registered new tserver with Master: 28ed4c6ac97348efbca01275baebff16 (127.23.115.3:46435)
I20260620 18:39:32.313601 1889 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.115.3@KRBTEST.COM'} at 127.23.115.3:34349
I20260620 18:39:32.317843 2078 heartbeater.cc:503] Master 127.23.115.62:45983 was elected leader, sending a full tablet report...
I20260620 18:39:32.320107 24012 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:39:32.338636 2080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:32.324759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54904 (local address 127.23.115.62:45983)
0620 18:39:32.325704 (+ 945us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:32.325711 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:32.325727 (+ 16us) server_negotiation.cc:407] Connection header received
0620 18:39:32.325784 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:32.325789 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:32.325860 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:32.325963 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:32.327680 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:32.328803 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:32.330303 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:32.330902 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:32.331034 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:32.333291 (+ 2257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:32.333298 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:32.333303 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:32.333343 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:32.336150 (+ 2807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:32.336756 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:32.336762 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:32.336766 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:32.336884 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:32.337225 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:32.337230 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:32.337234 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:32.337556 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:32.337791 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:32.338117 (+ 326us) server_negotiation.cc:299] Negotiation successful
0620 18:39:32.338324 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":811}
I20260620 18:39:32.342885 1889 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54904:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:39:32.349591 1889 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:39:32.372267 2360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:32.362766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60846 (local address 127.23.115.1:42777)
0620 18:39:32.363492 (+ 726us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:32.363500 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:32.364641 (+ 1141us) server_negotiation.cc:407] Connection header received
0620 18:39:32.364850 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:32.364857 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:32.365064 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:32.365234 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:39:32.365874 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:32.367227 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:32.369292 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:32.370674 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:32.370816 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:32.371088 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:32.371818 (+ 730us) server_negotiation.cc:299] Negotiation successful
0620 18:39:32.371972 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":661,"threads_started":1}
I20260620 18:39:32.372272 2362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:32.364112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32914 (local address 127.23.115.2:42135)
0620 18:39:32.365358 (+ 1246us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:32.365366 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:32.365392 (+ 26us) server_negotiation.cc:407] Connection header received
0620 18:39:32.365476 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:32.365481 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:32.365633 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:32.365791 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:39:32.366270 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:32.367575 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:32.369919 (+ 2344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:32.370963 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:32.371084 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:32.371277 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:32.371794 (+ 517us) server_negotiation.cc:299] Negotiation successful
0620 18:39:32.371958 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1101,"thread_start_us":1002,"threads_started":1}
I20260620 18:39:32.372977 2359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:32.362511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49390 (local address 127.23.115.3:46435)
0620 18:39:32.363136 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:32.363144 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:32.363391 (+ 247us) server_negotiation.cc:407] Connection header received
0620 18:39:32.363610 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:32.363619 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:32.363872 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:32.364040 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:39:32.365062 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:32.366633 (+ 1571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:32.369545 (+ 2912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:32.370858 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:32.370992 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:32.371191 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:32.372538 (+ 1347us) server_negotiation.cc:299] Negotiation successful
0620 18:39:32.372713 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":312,"threads_started":1}
I20260620 18:39:32.372363 2357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:32.362124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.3:46435 (local address 127.0.0.1:49390)
0620 18:39:32.363077 (+ 953us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:32.363113 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:32.363344 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:32.364362 (+ 1018us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:32.364370 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:32.364403 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:39:32.364860 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:32.364871 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:32.366765 (+ 1894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:32.366771 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:32.369390 (+ 2619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:32.369404 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:32.371332 (+ 1928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:32.371338 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:32.371571 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:32.371608 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 18:39:32.371678 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 18:39:32.371887 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":681,"mutex_wait_us":282,"thread_start_us":759,"threads_started":1}
I20260620 18:39:32.372370 2361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:32.363879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.2:42135 (local address 127.0.0.1:32914)
0620 18:39:32.364371 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:32.364387 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:32.364501 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:32.365793 (+ 1292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:32.365797 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:32.365819 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:39:32.366124 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:32.366133 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:32.367738 (+ 1605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:32.367743 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:32.369784 (+ 2041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:32.369794 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:32.371425 (+ 1631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:32.371431 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:32.371611 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:32.371643 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 18:39:32.371732 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 18:39:32.371914 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"mutex_wait_us":66,"thread_start_us":235,"threads_started":1}
I20260620 18:39:32.373534 2358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:32.362205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.1:42777 (local address 127.0.0.1:60846)
0620 18:39:32.363802 (+ 1597us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:32.363819 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:32.364672 (+ 853us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:32.365362 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:32.365366 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:32.365389 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:39:32.365696 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:32.365706 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:32.367382 (+ 1676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:32.367387 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:32.369141 (+ 1754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:32.369152 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:32.371213 (+ 2061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:32.371219 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:32.371433 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:32.371552 (+ 119us) client_negotiation.cc:769] Sending connection context
0620 18:39:32.371678 (+ 126us) client_negotiation.cc:240] Negotiation successful
0620 18:39:32.372696 (+ 1018us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1482,"mutex_wait_us":35,"thread_start_us":1346,"threads_started":1}
I20260620 18:39:32.377002 2013 tablet_service.cc:1467] Processing CreateTablet for tablet 3bc1a055eb34429db7dd90f3dc85c53f (DEFAULT_TABLE table=test-table [id=0dc9b5b104ee4f0e96e4ebab642ee554]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:39:32.377049 2147 tablet_service.cc:1467] Processing CreateTablet for tablet 3bc1a055eb34429db7dd90f3dc85c53f (DEFAULT_TABLE table=test-table [id=0dc9b5b104ee4f0e96e4ebab642ee554]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:39:32.377327 2280 tablet_service.cc:1467] Processing CreateTablet for tablet 3bc1a055eb34429db7dd90f3dc85c53f (DEFAULT_TABLE table=test-table [id=0dc9b5b104ee4f0e96e4ebab642ee554]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:39:32.378230 2013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bc1a055eb34429db7dd90f3dc85c53f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:39:32.378419 2147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bc1a055eb34429db7dd90f3dc85c53f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:39:32.378417 2280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bc1a055eb34429db7dd90f3dc85c53f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:39:32.390120 2363 tablet_bootstrap.cc:492] T 3bc1a055eb34429db7dd90f3dc85c53f P 6103605c1ae34c638dc21ba5a8b8e0a9: Bootstrap starting.
I20260620 18:39:32.393576 2365 tablet_bootstrap.cc:492] T 3bc1a055eb34429db7dd90f3dc85c53f P 28ed4c6ac97348efbca01275baebff16: Bootstrap starting.
I20260620 18:39:32.393827 2364 tablet_bootstrap.cc:492] T 3bc1a055eb34429db7dd90f3dc85c53f P 4c86ce0e45f64483bcf286c12b1bbd2f: Bootstrap starting.
I20260620 18:39:32.394568 2363 tablet_bootstrap.cc:654] T 3bc1a055eb34429db7dd90f3dc85c53f P 6103605c1ae34c638dc21ba5a8b8e0a9: Neither blocks nor log segments found. Creating new log.
I20260620 18:39:32.395905 2363 log.cc:824] T 3bc1a055eb34429db7dd90f3dc85c53f P 6103605c1ae34c638dc21ba5a8b8e0a9: Log is configured to *not* fsync() on all Append() calls
I20260620 18:39:32.397171 2365 tablet_bootstrap.cc:654] T 3bc1a055eb34429db7dd90f3dc85c53f P 28ed4c6ac97348efbca01275baebff16: Neither blocks nor log segments found. Creating new log.
I20260620 18:39:32.397459 2364 tablet_bootstrap.cc:654] T 3bc1a055eb34429db7dd90f3dc85c53f P 4c86ce0e45f64483bcf286c12b1bbd2f: Neither blocks nor log segments found. Creating new log.
I20260620 18:39:32.398412 2364 log.cc:824] T 3bc1a055eb34429db7dd90f3dc85c53f P 4c86ce0e45f64483bcf286c12b1bbd2f: Log is configured to *not* fsync() on all Append() calls
I20260620 18:39:32.398442 2365 log.cc:824] T 3bc1a055eb34429db7dd90f3dc85c53f P 28ed4c6ac97348efbca01275baebff16: Log is configured to *not* fsync() on all Append() calls
I20260620 18:39:32.399418 2363 tablet_bootstrap.cc:492] T 3bc1a055eb34429db7dd90f3dc85c53f P 6103605c1ae34c638dc21ba5a8b8e0a9: No bootstrap required, opened a new log
I20260620 18:39:32.399679 2363 ts_tablet_manager.cc:1397] T 3bc1a055eb34429db7dd90f3dc85c53f P 6103605c1ae34c638dc21ba5a8b8e0a9: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260620 18:39:32.400985 2365 tablet_bootstrap.cc:492] T 3bc1a055eb34429db7dd90f3dc85c53f P 28ed4c6ac97348efbca01275baebff16: No bootstrap required, opened a new log
I20260620 18:39:32.401219 2365 ts_tablet_manager.cc:1397] T 3bc1a055eb34429db7dd90f3dc85c53f P 28ed4c6ac97348efbca01275baebff16: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.001s
I20260620 18:39:32.401240 2364 tablet_bootstrap.cc:492] T 3bc1a055eb34429db7dd90f3dc85c53f P 4c86ce0e45f64483bcf286c12b1bbd2f: No bootstrap required, opened a new log
I20260620 18:39:32.401468 2364 ts_tablet_manager.cc:1397] T 3bc1a055eb34429db7dd90f3dc85c53f P 4c86ce0e45f64483bcf286c12b1bbd2f: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.002s
I20260620 18:39:32.408411 2365 raft_consensus.cc:348] T 3bc1a055eb34429db7dd90f3dc85c53f P 28ed4c6ac97348efbca01275baebff16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28ed4c6ac97348efbca01275baebff16" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 46435 } } peers { permanent_uuid: "6103605c1ae34c638dc21ba5a8b8e0a9" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 42777 } } peers { permanent_uuid: "4c86ce0e45f64483bcf286c12b1bbd2f" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 42135 } }
I20260620 18:39:32.408763 2365 raft_consensus.cc:374] T 3bc1a055eb34429db7dd90f3dc85c53f P 28ed4c6ac97348efbca01275baebff16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:39:32.408876 2365 raft_consensus.cc:729] T 3bc1a055eb34429db7dd90f3dc85c53f P 28ed4c6ac97348efbca01275baebff16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28ed4c6ac97348efbca01275baebff16, State: Initialized, Role: FOLLOWER
I20260620 18:39:32.409446 2365 consensus_queue.cc:260] T 3bc1a055eb34429db7dd90f3dc85c53f P 28ed4c6ac97348efbca01275baebff16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28ed4c6ac97348efbca01275baebff16" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 46435 } } peers { permanent_uuid: "6103605c1ae34c638dc21ba5a8b8e0a9" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 42777 } } peers { permanent_uuid: "4c86ce0e45f64483bcf286c12b1bbd2f" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 42135 } }
I20260620 18:39:32.409956 2363 raft_consensus.cc:348] T 3bc1a055eb34429db7dd90f3dc85c53f P 6103605c1ae34c638dc21ba5a8b8e0a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28ed4c6ac97348efbca01275baebff16" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 46435 } } peers { permanent_uuid: "6103605c1ae34c638dc21ba5a8b8e0a9" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 42777 } } peers { permanent_uuid: "4c86ce0e45f64483bcf286c12b1bbd2f" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 42135 } }
I20260620 18:39:32.410420 2363 raft_consensus.cc:374] T 3bc1a055eb34429db7dd90f3dc85c53f P 6103605c1ae34c638dc21ba5a8b8e0a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:39:32.410490 2345 heartbeater.cc:503] Master 127.23.115.62:45983 was elected leader, sending a full tablet report...
I20260620 18:39:32.410529 2363 raft_consensus.cc:729] T 3bc1a055eb34429db7dd90f3dc85c53f P 6103605c1ae34c638dc21ba5a8b8e0a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6103605c1ae34c638dc21ba5a8b8e0a9, State: Initialized, Role: FOLLOWER
I20260620 18:39:32.411135 2365 ts_tablet_manager.cc:1428] T 3bc1a055eb34429db7dd90f3dc85c53f P 28ed4c6ac97348efbca01275baebff16: Time spent starting tablet: real 0.010s user 0.008s sys 0.003s
I20260620 18:39:32.411109 2363 consensus_queue.cc:260] T 3bc1a055eb34429db7dd90f3dc85c53f P 6103605c1ae34c638dc21ba5a8b8e0a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28ed4c6ac97348efbca01275baebff16" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 46435 } } peers { permanent_uuid: "6103605c1ae34c638dc21ba5a8b8e0a9" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 42777 } } peers { permanent_uuid: "4c86ce0e45f64483bcf286c12b1bbd2f" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 42135 } }
I20260620 18:39:32.412765 2363 ts_tablet_manager.cc:1428] T 3bc1a055eb34429db7dd90f3dc85c53f P 6103605c1ae34c638dc21ba5a8b8e0a9: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260620 18:39:32.412760 2364 raft_consensus.cc:348] T 3bc1a055eb34429db7dd90f3dc85c53f P 4c86ce0e45f64483bcf286c12b1bbd2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28ed4c6ac97348efbca01275baebff16" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 46435 } } peers { permanent_uuid: "6103605c1ae34c638dc21ba5a8b8e0a9" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 42777 } } peers { permanent_uuid: "4c86ce0e45f64483bcf286c12b1bbd2f" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 42135 } }
I20260620 18:39:32.413210 2364 raft_consensus.cc:374] T 3bc1a055eb34429db7dd90f3dc85c53f P 4c86ce0e45f64483bcf286c12b1bbd2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:39:32.413341 2364 raft_consensus.cc:729] T 3bc1a055eb34429db7dd90f3dc85c53f P 4c86ce0e45f64483bcf286c12b1bbd2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c86ce0e45f64483bcf286c12b1bbd2f, State: Initialized, Role: FOLLOWER
I20260620 18:39:32.413992 2364 consensus_queue.cc:260] T 3bc1a055eb34429db7dd90f3dc85c53f P 4c86ce0e45f64483bcf286c12b1bbd2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28ed4c6ac97348efbca01275baebff16" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 46435 } } peers { permanent_uuid: "6103605c1ae34c638dc21ba5a8b8e0a9" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 42777 } } peers { permanent_uuid: "4c86ce0e45f64483bcf286c12b1bbd2f" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 42135 } }
I20260620 18:39:32.415146 2212 heartbeater.cc:503] Master 127.23.115.62:45983 was elected leader, sending a full tablet report...
I20260620 18:39:32.415733 2364 ts_tablet_manager.cc:1428] T 3bc1a055eb34429db7dd90f3dc85c53f P 4c86ce0e45f64483bcf286c12b1bbd2f: Time spent starting tablet: real 0.014s user 0.011s sys 0.003s
I20260620 18:39:32.425129 2371 raft_consensus.cc:482] T 3bc1a055eb34429db7dd90f3dc85c53f P 4c86ce0e45f64483bcf286c12b1bbd2f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:39:32.425414 2371 raft_consensus.cc:504] T 3bc1a055eb34429db7dd90f3dc85c53f P 4c86ce0e45f64483bcf286c12b1bbd2f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28ed4c6ac97348efbca01275baebff16" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 46435 } } peers { permanent_uuid: "6103605c1ae34c638dc21ba5a8b8e0a9" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 42777 } } peers { permanent_uuid: "4c86ce0e45f64483bcf286c12b1bbd2f" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 42135 } }
I20260620 18:39:32.427863 2371 leader_election.cc:288] T 3bc1a055eb34429db7dd90f3dc85c53f P 4c86ce0e45f64483bcf286c12b1bbd2f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28ed4c6ac97348efbca01275baebff16 (127.23.115.3:46435), 6103605c1ae34c638dc21ba5a8b8e0a9 (127.23.115.1:42777)
I20260620 18:39:32.434532 2359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:32.428215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.2:33065 (local address 127.23.115.3:46435)
0620 18:39:32.428470 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:32.428477 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:32.429287 (+ 810us) server_negotiation.cc:407] Connection header received
0620 18:39:32.429452 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:32.429458 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:32.429524 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:32.429588 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:39:32.430150 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:32.430870 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:32.432780 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:32.433517 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:32.433644 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:32.433698 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:32.434160 (+ 462us) server_negotiation.cc:299] Negotiation successful
0620 18:39:32.434287 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260620 18:39:32.434638 2372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:32.428085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.3:46435 (local address 127.23.115.2:33065)
0620 18:39:32.429178 (+ 1093us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:32.429196 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:32.429320 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:32.429621 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:32.429625 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:32.429646 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:39:32.430017 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:32.430029 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:32.431010 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:32.431015 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:32.432629 (+ 1614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:32.432640 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:32.433733 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:32.433739 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:32.433905 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:32.433943 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 18:39:32.434131 (+ 188us) client_negotiation.cc:240] Negotiation successful
0620 18:39:32.434228 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":942,"mutex_wait_us":87,"thread_start_us":689,"threads_started":1}
I20260620 18:39:32.434698 2360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:32.428513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.2:40945 (local address 127.23.115.1:42777)
0620 18:39:32.428724 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:32.428730 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:32.429324 (+ 594us) server_negotiation.cc:407] Connection header received
0620 18:39:32.429474 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:32.429478 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:32.429534 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:32.429609 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:39:32.430221 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:32.430916 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:32.432792 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:32.433794 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:32.433947 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:32.434089 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:32.434354 (+ 265us) server_negotiation.cc:299] Negotiation successful
0620 18:39:32.434484 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 18:39:32.434890 2373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:32.428350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.1:42777 (local address 127.23.115.2:40945)
0620 18:39:32.429216 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0620 18:39:32.429229 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 18:39:32.429357 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:39:32.429632 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:39:32.429636 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:39:32.429658 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:39:32.430075 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:32.430085 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:32.431054 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:32.431058 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:32.432634 (+ 1576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:39:32.432646 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:39:32.434021 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:39:32.434026 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:39:32.434179 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:32.434209 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 18:39:32.434318 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 18:39:32.434442 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":772,"mutex_wait_us":22,"thread_start_us":592,"threads_started":1}
I20260620 18:39:32.435778 2300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bc1a055eb34429db7dd90f3dc85c53f" candidate_uuid: "4c86ce0e45f64483bcf286c12b1bbd2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28ed4c6ac97348efbca01275baebff16" is_pre_election: true
I20260620 18:39:32.435786 2033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bc1a055eb34429db7dd90f3dc85c53f" candidate_uuid: "4c86ce0e45f64483bcf286c12b1bbd2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6103605c1ae34c638dc21ba5a8b8e0a9" is_pre_election: true
I20260620 18:39:32.436237 2300 raft_consensus.cc:2454] T 3bc1a055eb34429db7dd90f3dc85c53f P 28ed4c6ac97348efbca01275baebff16 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c86ce0e45f64483bcf286c12b1bbd2f in term 0.
I20260620 18:39:32.436239 2033 raft_consensus.cc:2454] T 3bc1a055eb34429db7dd90f3dc85c53f P 6103605c1ae34c638dc21ba5a8b8e0a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c86ce0e45f64483bcf286c12b1bbd2f in term 0.
I20260620 18:39:32.436817 2100 leader_election.cc:302] T 3bc1a055eb34429db7dd90f3dc85c53f P 4c86ce0e45f64483bcf286c12b1bbd2f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c86ce0e45f64483bcf286c12b1bbd2f, 6103605c1ae34c638dc21ba5a8b8e0a9; no voters:
I20260620 18:39:32.437156 2371 raft_consensus.cc:2781] T 3bc1a055eb34429db7dd90f3dc85c53f P 4c86ce0e45f64483bcf286c12b1bbd2f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:39:32.437286 2371 raft_consensus.cc:482] T 3bc1a055eb34429db7dd90f3dc85c53f P 4c86ce0e45f64483bcf286c12b1bbd2f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:39:32.437381 2371 raft_consensus.cc:3037] T 3bc1a055eb34429db7dd90f3dc85c53f P 4c86ce0e45f64483bcf286c12b1bbd2f [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:39:32.439548 2371 raft_consensus.cc:504] T 3bc1a055eb34429db7dd90f3dc85c53f P 4c86ce0e45f64483bcf286c12b1bbd2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28ed4c6ac97348efbca01275baebff16" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 46435 } } peers { permanent_uuid: "6103605c1ae34c638dc21ba5a8b8e0a9" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 42777 } } peers { permanent_uuid: "4c86ce0e45f64483bcf286c12b1bbd2f" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 42135 } }
I20260620 18:39:32.440102 2371 leader_election.cc:288] T 3bc1a055eb34429db7dd90f3dc85c53f P 4c86ce0e45f64483bcf286c12b1bbd2f [CANDIDATE]: Term 1 election: Requested vote from peers 28ed4c6ac97348efbca01275baebff16 (127.23.115.3:46435), 6103605c1ae34c638dc21ba5a8b8e0a9 (127.23.115.1:42777)
I20260620 18:39:32.440498 2300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bc1a055eb34429db7dd90f3dc85c53f" candidate_uuid: "4c86ce0e45f64483bcf286c12b1bbd2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28ed4c6ac97348efbca01275baebff16"
I20260620 18:39:32.440524 2033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bc1a055eb34429db7dd90f3dc85c53f" candidate_uuid: "4c86ce0e45f64483bcf286c12b1bbd2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6103605c1ae34c638dc21ba5a8b8e0a9"
I20260620 18:39:32.440668 2300 raft_consensus.cc:3037] T 3bc1a055eb34429db7dd90f3dc85c53f P 28ed4c6ac97348efbca01275baebff16 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:39:32.440685 2033 raft_consensus.cc:3037] T 3bc1a055eb34429db7dd90f3dc85c53f P 6103605c1ae34c638dc21ba5a8b8e0a9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:39:32.442919 2033 raft_consensus.cc:2454] T 3bc1a055eb34429db7dd90f3dc85c53f P 6103605c1ae34c638dc21ba5a8b8e0a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c86ce0e45f64483bcf286c12b1bbd2f in term 1.
I20260620 18:39:32.443101 2300 raft_consensus.cc:2454] T 3bc1a055eb34429db7dd90f3dc85c53f P 28ed4c6ac97348efbca01275baebff16 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c86ce0e45f64483bcf286c12b1bbd2f in term 1.
I20260620 18:39:32.443421 2100 leader_election.cc:302] T 3bc1a055eb34429db7dd90f3dc85c53f P 4c86ce0e45f64483bcf286c12b1bbd2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c86ce0e45f64483bcf286c12b1bbd2f, 6103605c1ae34c638dc21ba5a8b8e0a9; no voters:
I20260620 18:39:32.443718 2371 raft_consensus.cc:2781] T 3bc1a055eb34429db7dd90f3dc85c53f P 4c86ce0e45f64483bcf286c12b1bbd2f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:39:32.444329 2371 raft_consensus.cc:686] T 3bc1a055eb34429db7dd90f3dc85c53f P 4c86ce0e45f64483bcf286c12b1bbd2f [term 1 LEADER]: Becoming Leader. State: Replica: 4c86ce0e45f64483bcf286c12b1bbd2f, State: Running, Role: LEADER
I20260620 18:39:32.444880 2371 consensus_queue.cc:237] T 3bc1a055eb34429db7dd90f3dc85c53f P 4c86ce0e45f64483bcf286c12b1bbd2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28ed4c6ac97348efbca01275baebff16" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 46435 } } peers { permanent_uuid: "6103605c1ae34c638dc21ba5a8b8e0a9" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 42777 } } peers { permanent_uuid: "4c86ce0e45f64483bcf286c12b1bbd2f" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 42135 } }
I20260620 18:39:32.449144 1888 catalog_manager.cc:5495] T 3bc1a055eb34429db7dd90f3dc85c53f P 4c86ce0e45f64483bcf286c12b1bbd2f reported cstate change: term changed from 0 to 1, leader changed from <none> to 4c86ce0e45f64483bcf286c12b1bbd2f (127.23.115.2). New cstate: current_term: 1 leader_uuid: "4c86ce0e45f64483bcf286c12b1bbd2f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28ed4c6ac97348efbca01275baebff16" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 46435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6103605c1ae34c638dc21ba5a8b8e0a9" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 42777 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c86ce0e45f64483bcf286c12b1bbd2f" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 42135 } health_report { overall_health: HEALTHY } } }
I20260620 18:39:32.476274 2362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:32.470804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32920 (local address 127.23.115.2:42135)
0620 18:39:32.471016 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:32.471020 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:32.471037 (+ 17us) server_negotiation.cc:407] Connection header received
0620 18:39:32.471217 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:32.471220 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:32.471304 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:32.471425 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:39:32.471814 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:32.472553 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:32.473873 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:32.474258 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:32.474358 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:32.475172 (+ 814us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:39:32.475679 (+ 507us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:39:32.475801 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:32.475984 (+ 183us) server_negotiation.cc:299] Negotiation successful
0620 18:39:32.476057 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 18:39:32.484963 2033 raft_consensus.cc:1261] T 3bc1a055eb34429db7dd90f3dc85c53f P 6103605c1ae34c638dc21ba5a8b8e0a9 [term 1 FOLLOWER]: Refusing update from remote peer 4c86ce0e45f64483bcf286c12b1bbd2f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:39:32.485460 2300 raft_consensus.cc:1261] T 3bc1a055eb34429db7dd90f3dc85c53f P 28ed4c6ac97348efbca01275baebff16 [term 1 FOLLOWER]: Refusing update from remote peer 4c86ce0e45f64483bcf286c12b1bbd2f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:39:32.485894 2374 consensus_queue.cc:1035] T 3bc1a055eb34429db7dd90f3dc85c53f P 4c86ce0e45f64483bcf286c12b1bbd2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6103605c1ae34c638dc21ba5a8b8e0a9" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 42777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:39:32.486270 2371 consensus_queue.cc:1035] T 3bc1a055eb34429db7dd90f3dc85c53f P 4c86ce0e45f64483bcf286c12b1bbd2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "28ed4c6ac97348efbca01275baebff16" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 46435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:39:32.496446 2377 mvcc.cc:205] Tried to move back new op lower bound from 7298993244088676352 to 7298993243938181120. Current Snapshot: MvccSnapshot[applied={T|T < 7298993244088676352}]
I20260620 18:39:32.512055 1888 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54904:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:39:32.512377 1888 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54904:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:39:32.516441 1888 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e9bb4fa99e964acba47fadd442ee8f14: Sending DeleteTablet for 3 replicas of tablet 3bc1a055eb34429db7dd90f3dc85c53f
I20260620 18:39:32.517510 2280 tablet_service.cc:1514] Processing DeleteTablet for tablet 3bc1a055eb34429db7dd90f3dc85c53f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:39:32 UTC) from {username='kudu', principal='kudu/127.23.115.62@KRBTEST.COM'} at 127.0.0.1:49390
I20260620 18:39:32.517686 2147 tablet_service.cc:1514] Processing DeleteTablet for tablet 3bc1a055eb34429db7dd90f3dc85c53f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:39:32 UTC) from {username='kudu', principal='kudu/127.23.115.62@KRBTEST.COM'} at 127.0.0.1:32914
I20260620 18:39:32.518096 2013 tablet_service.cc:1514] Processing DeleteTablet for tablet 3bc1a055eb34429db7dd90f3dc85c53f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:39:32 UTC) from {username='kudu', principal='kudu/127.23.115.62@KRBTEST.COM'} at 127.0.0.1:60846
I20260620 18:39:32.518448 2388 tablet_replica.cc:330] T 3bc1a055eb34429db7dd90f3dc85c53f P 4c86ce0e45f64483bcf286c12b1bbd2f: stopping tablet replica
I20260620 18:39:32.518802 2387 tablet_replica.cc:330] T 3bc1a055eb34429db7dd90f3dc85c53f P 28ed4c6ac97348efbca01275baebff16: stopping tablet replica
I20260620 18:39:32.518880 2388 raft_consensus.cc:2229] T 3bc1a055eb34429db7dd90f3dc85c53f P 4c86ce0e45f64483bcf286c12b1bbd2f [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:39:32.519196 2387 raft_consensus.cc:2229] T 3bc1a055eb34429db7dd90f3dc85c53f P 28ed4c6ac97348efbca01275baebff16 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:39:32.519542 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 1950
I20260620 18:39:32.519582 2388 raft_consensus.cc:2258] T 3bc1a055eb34429db7dd90f3dc85c53f P 4c86ce0e45f64483bcf286c12b1bbd2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:39:32.519688 2387 raft_consensus.cc:2258] T 3bc1a055eb34429db7dd90f3dc85c53f P 28ed4c6ac97348efbca01275baebff16 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:39:32.521975 2387 ts_tablet_manager.cc:1905] T 3bc1a055eb34429db7dd90f3dc85c53f P 28ed4c6ac97348efbca01275baebff16: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:39:32.524225 2389 tablet_replica.cc:330] T 3bc1a055eb34429db7dd90f3dc85c53f P 6103605c1ae34c638dc21ba5a8b8e0a9: stopping tablet replica
I20260620 18:39:32.525261 2389 raft_consensus.cc:2229] T 3bc1a055eb34429db7dd90f3dc85c53f P 6103605c1ae34c638dc21ba5a8b8e0a9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:39:32.525537 2388 ts_tablet_manager.cc:1905] T 3bc1a055eb34429db7dd90f3dc85c53f P 4c86ce0e45f64483bcf286c12b1bbd2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:39:32.526508 2389 raft_consensus.cc:2258] T 3bc1a055eb34429db7dd90f3dc85c53f P 6103605c1ae34c638dc21ba5a8b8e0a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:39:32.529152 2387 ts_tablet_manager.cc:1918] T 3bc1a055eb34429db7dd90f3dc85c53f P 28ed4c6ac97348efbca01275baebff16: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:39:32.529343 2387 log.cc:1196] T 3bc1a055eb34429db7dd90f3dc85c53f P 28ed4c6ac97348efbca01275baebff16: Deleting WAL directory at /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980676253068-24012-0/minicluster-data/ts-2/wal/wals/3bc1a055eb34429db7dd90f3dc85c53f
I20260620 18:39:32.530205 2387 ts_tablet_manager.cc:1939] T 3bc1a055eb34429db7dd90f3dc85c53f P 28ed4c6ac97348efbca01275baebff16: Deleting consensus metadata
I20260620 18:39:32.532253 1875 catalog_manager.cc:4841] TS 28ed4c6ac97348efbca01275baebff16 (127.23.115.3:46435): tablet 3bc1a055eb34429db7dd90f3dc85c53f (table test-table [id=0dc9b5b104ee4f0e96e4ebab642ee554]) successfully deleted
I20260620 18:39:32.534165 2388 ts_tablet_manager.cc:1918] T 3bc1a055eb34429db7dd90f3dc85c53f P 4c86ce0e45f64483bcf286c12b1bbd2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:39:32.534369 2388 log.cc:1196] T 3bc1a055eb34429db7dd90f3dc85c53f P 4c86ce0e45f64483bcf286c12b1bbd2f: Deleting WAL directory at /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980676253068-24012-0/minicluster-data/ts-1/wal/wals/3bc1a055eb34429db7dd90f3dc85c53f
I20260620 18:39:32.535015 2388 ts_tablet_manager.cc:1939] T 3bc1a055eb34429db7dd90f3dc85c53f P 4c86ce0e45f64483bcf286c12b1bbd2f: Deleting consensus metadata
I20260620 18:39:32.536432 2389 ts_tablet_manager.cc:1905] T 3bc1a055eb34429db7dd90f3dc85c53f P 6103605c1ae34c638dc21ba5a8b8e0a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:39:32.536573 1873 catalog_manager.cc:4841] TS 4c86ce0e45f64483bcf286c12b1bbd2f (127.23.115.2:42135): tablet 3bc1a055eb34429db7dd90f3dc85c53f (table test-table [id=0dc9b5b104ee4f0e96e4ebab642ee554]) successfully deleted
I20260620 18:39:32.543639 2389 ts_tablet_manager.cc:1918] T 3bc1a055eb34429db7dd90f3dc85c53f P 6103605c1ae34c638dc21ba5a8b8e0a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:39:32.543823 2389 log.cc:1196] T 3bc1a055eb34429db7dd90f3dc85c53f P 6103605c1ae34c638dc21ba5a8b8e0a9: Deleting WAL directory at /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980676253068-24012-0/minicluster-data/ts-0/wal/wals/3bc1a055eb34429db7dd90f3dc85c53f
I20260620 18:39:32.544755 2389 ts_tablet_manager.cc:1939] T 3bc1a055eb34429db7dd90f3dc85c53f P 6103605c1ae34c638dc21ba5a8b8e0a9: Deleting consensus metadata
Jun 20 18:39:32 dist-test-slave-d645 krb5kdc[1844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980770, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.115.1@KRBTEST.COM
I20260620 18:39:32.546860 1873 catalog_manager.cc:4841] TS 6103605c1ae34c638dc21ba5a8b8e0a9 (127.23.115.1:42777): tablet 3bc1a055eb34429db7dd90f3dc85c53f (table test-table [id=0dc9b5b104ee4f0e96e4ebab642ee554]) successfully deleted
I20260620 18:39:32.553294 2360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:32.520628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60852 (local address 127.23.115.1:42777)
0620 18:39:32.520788 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:32.520795 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:32.535029 (+ 14234us) server_negotiation.cc:407] Connection header received
0620 18:39:32.539313 (+ 4284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:32.539318 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:32.539383 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:32.539481 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:32.540664 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:32.541414 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:32.542797 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:32.543279 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:32.543404 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:32.547859 (+ 4455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:32.547870 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:32.547884 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:32.547916 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:32.550603 (+ 2687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:32.551381 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:32.551387 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:32.551390 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:32.551504 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:32.551853 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:32.551856 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:32.551859 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:32.552202 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:32.552482 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:32.552765 (+ 283us) server_negotiation.cc:299] Negotiation successful
0620 18:39:32.552969 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 18:39:32.553936 2073 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:39:32.643036 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 1950
I20260620 18:39:32.660305 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 2084
Jun 20 18:39:32 dist-test-slave-d645 krb5kdc[1844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980770, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.115.2@KRBTEST.COM
I20260620 18:39:32.674654 2362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:32.661107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32924 (local address 127.23.115.2:42135)
0620 18:39:32.661309 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:32.661315 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:32.661329 (+ 14us) server_negotiation.cc:407] Connection header received
0620 18:39:32.661379 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:32.661382 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:32.661464 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:32.661555 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:32.662740 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:32.663577 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:32.664767 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:32.665197 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:32.665313 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:32.669169 (+ 3856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:32.669182 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:32.669200 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:32.669237 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:32.672100 (+ 2863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:32.672698 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:32.672705 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:32.672707 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:32.672841 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:32.673171 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:32.673175 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:32.673181 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:32.673549 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:32.673856 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:32.674106 (+ 250us) server_negotiation.cc:299] Negotiation successful
0620 18:39:32.674329 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 18:39:32.675386 2207 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:39:32.788280 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 2084
I20260620 18:39:32.804477 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 2217
Jun 20 18:39:32 dist-test-slave-d645 krb5kdc[1844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980770, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.115.3@KRBTEST.COM
I20260620 18:39:32.819803 2359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:39:32.805236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49392 (local address 127.23.115.3:46435)
0620 18:39:32.805406 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 18:39:32.805410 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:39:32.805429 (+ 19us) server_negotiation.cc:407] Connection header received
0620 18:39:32.805583 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:39:32.805587 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:39:32.805656 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:39:32.805744 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:39:32.806984 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:32.807781 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:32.809076 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:39:32.809761 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:39:32.809868 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:39:32.814047 (+ 4179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:39:32.814061 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:39:32.814077 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:39:32.814118 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:39:32.816694 (+ 2576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:32.817320 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:32.817326 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:32.817329 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:32.817446 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:39:32.817790 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:39:32.817794 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:39:32.817796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:39:32.818133 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:39:32.818380 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0620 18:39:32.818903 (+ 523us) server_negotiation.cc:299] Negotiation successful
0620 18:39:32.819241 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 18:39:32.820484 2340 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:39:32.908854 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 2217
I20260620 18:39:32.921983 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 1857
I20260620 18:39:32.923337 1919 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:39:33.033249 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 1857
2026-06-20T18:39:33Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4085 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:39:33 dist-test-slave-d645 krb5kdc[2397](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:39:33 dist-test-slave-d645 krb5kdc[2397](info): set up 2 sockets
Jun 20 18:39:33 dist-test-slave-d645 krb5kdc[2397](info): commencing operation
krb5kdc: starting...
W20260620 18:39:34.075799 24012 mini_kdc.cc:121] Time spent starting KDC: real 1.002s user 0.004s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/krb5kdc/test-user.keytab.
Jun 20 18:39:34 dist-test-slave-d645 krb5kdc[2397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980774, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:39:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:39:34Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.23.115.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.23.115.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.23.115.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/krb5kdc/rangeradmin_127.23.115.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.23.115.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/krb5kdc/rangeradmin_127.23.115.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.23.115.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.23.115.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.23.115.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/krb5kdc/rangerlookup_127.23.115.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.23.115.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/krb5kdc/rangerlookup_127.23.115.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.23.115.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.23.115.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.115.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/krb5kdc/HTTP_127.23.115.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.23.115.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/krb5kdc/HTTP_127.23.115.20@KRBTEST.COM.keytab.
I20260620 18:39:34.391243 24012 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task6x5P9o/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/minicluster-data/postgres -l logfile start
2026-06-20 18:39:37.156 UTC [2427] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 18:39:37.156 UTC [2427] LOG: listening on IPv4 address "127.23.115.20", port 59515
2026-06-20 18:39:37.161 UTC [2427] LOG: listening on Unix socket "/tmp/.s.PGSQL.59515"
2026-06-20 18:39:37.183 UTC [2430] LOG: database system was shut down at 2026-06-20 18:39:35 UTC
2026-06-20 18:39:37.188 UTC [2427] LOG: database system is ready to accept connections
I20260620 18:39:38.095541 24012 mini_postgres.cc:96] Postgres bound to 59515
2026-06-20 18:39:38.112 UTC [2438] FATAL: database "slave" does not exist
127.23.115.20:59515 - accepting connections
I20260620 18:39:38.112766 24012 mini_ranger.cc:161] Starting Ranger...
I20260620 18:39:38.137544 24012 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 18:39:38.754098 24012 mini_ranger.cc:87] Created ranger Postgres database
I20260620 18:39:38.754212 24012 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/minicluster-data/ranger-admin
2026-06-20 18:39:39,086 [I] DB FLAVOR :POSTGRES
2026-06-20 18:39:39,086 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 18:39:39,087 [I] Checking connection..
2026-06-20 18:39:39,087 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6x5P9o/build/asan/bin/postgresql.jar:/tmp/dist-test-task6x5P9o/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.115.20:59515/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 18:39:39,356 [I] Checking connection passed.
2026-06-20 18:39:39,357 [I] --------- Verifying version history table ---------
2026-06-20 18:39:39,357 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6x5P9o/build/asan/bin/postgresql.jar:/tmp/dist-test-task6x5P9o/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.115.20:59515/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 18:39:39,618 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 18:39:39,619 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6x5P9o/build/asan/bin/postgresql.jar:/tmp/dist-test-task6x5P9o/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.115.20:59515/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 18:39:39,878 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 18:39:39,879 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 18:39:39,879 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6x5P9o/build/asan/bin/postgresql.jar:/tmp/dist-test-task6x5P9o/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.115.20:59515/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task6x5P9o/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 18:39:40.144 UTC [2502] WARNING: there is no transaction in progress
2026-06-20 18:39:40,150 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 18:39:40,150 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6x5P9o/build/asan/bin/postgresql.jar:/tmp/dist-test-task6x5P9o/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.115.20:59515/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 18:39:40,421 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 18:39:40,421 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 18:39:40,422 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6x5P9o/build/asan/bin/postgresql.jar:/tmp/dist-test-task6x5P9o/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.115.20:59515/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 18:39:40,666 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6x5P9o/build/asan/bin/postgresql.jar:/tmp/dist-test-task6x5P9o/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.115.20:59515/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 18:39:40,905 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6x5P9o/build/asan/bin/postgresql.jar:/tmp/dist-test-task6x5P9o/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.115.20:59515/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-d645.c.gcp-upstream.internal','N') ;"
2026-06-20 18:39:41,155 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6x5P9o/build/asan/bin/postgresql.jar:/tmp/dist-test-task6x5P9o/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.115.20:59515/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 18:39:41,394 [I] Table x_portal_user does not exist in database ranger
2026-06-20 18:39:41,394 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6x5P9o/build/asan/bin/postgresql.jar:/tmp/dist-test-task6x5P9o/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.115.20:59515/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 18:39:41,661 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 18:39:41,662 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6x5P9o/build/asan/bin/postgresql.jar:/tmp/dist-test-task6x5P9o/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.115.20:59515/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 18:39:41,902 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6x5P9o/build/asan/bin/postgresql.jar:/tmp/dist-test-task6x5P9o/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.115.20:59515/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 18:39:42,154 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 18:39:42,155 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6x5P9o/build/asan/bin/postgresql.jar:/tmp/dist-test-task6x5P9o/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.115.20:59515/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task6x5P9o/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 18:39:43.153 UTC [2610] WARNING: there is no transaction in progress
2026-06-20 18:39:43.171 UTC [2610] WARNING: there is no transaction in progress
2026-06-20 18:39:43.189 UTC [2610] WARNING: there is no transaction in progress
2026-06-20 18:39:43.207 UTC [2610] WARNING: there is no transaction in progress
2026-06-20 18:39:43.227 UTC [2610] WARNING: there is no transaction in progress
2026-06-20 18:39:43.250 UTC [2610] WARNING: there is no transaction in progress
2026-06-20 18:39:43.313 UTC [2610] WARNING: there is no transaction in progress
2026-06-20 18:39:43.321 UTC [2610] WARNING: there is no transaction in progress
2026-06-20 18:39:43.337 UTC [2610] WARNING: there is no transaction in progress
2026-06-20 18:39:43.348 UTC [2610] WARNING: there is no transaction in progress
2026-06-20 18:39:43.359 UTC [2610] WARNING: there is no transaction in progress
2026-06-20 18:39:43.375 UTC [2610] WARNING: there is no transaction in progress
2026-06-20 18:39:43.386 UTC [2610] WARNING: there is no transaction in progress
2026-06-20 18:39:43.393 UTC [2610] WARNING: there is no transaction in progress
2026-06-20 18:39:43.400 UTC [2610] WARNING: there is no transaction in progress
2026-06-20 18:39:43.407 UTC [2610] WARNING: there is no transaction in progress
2026-06-20 18:39:44,101 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 18:39:44,102 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6x5P9o/build/asan/bin/postgresql.jar:/tmp/dist-test-task6x5P9o/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.115.20:59515/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 18:39:44,329 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 18:39:44,329 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6x5P9o/build/asan/bin/postgresql.jar:/tmp/dist-test-task6x5P9o/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.115.20:59515/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 18:39:44,578 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 18:39:44,578 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6x5P9o/build/asan/bin/postgresql.jar:/tmp/dist-test-task6x5P9o/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.115.20:59515/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 18:39:44,855 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 18:39:44,855 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6x5P9o/build/asan/bin/postgresql.jar:/tmp/dist-test-task6x5P9o/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.115.20:59515/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 18:39:45,087 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6x5P9o/build/asan/bin/postgresql.jar:/tmp/dist-test-task6x5P9o/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.115.20:59515/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 18:39:45,342 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6x5P9o/build/asan/bin/postgresql.jar:/tmp/dist-test-task6x5P9o/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.115.20:59515/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-d645.c.gcp-upstream.internal';"
2026-06-20 18:39:45,573 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 18:39:45,573 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6x5P9o/build/asan/bin/postgresql.jar:/tmp/dist-test-task6x5P9o/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.115.20:59515/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 18:39:45,825 [I] DB_PATCHES have already been applied
I20260620 18:39:45.832012 24012 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/minicluster-data/ranger-admin:/tmp/dist-test-task6x5P9o/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task6x5P9o/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task6x5P9o/build/asan/bin/postgresql.jar:/tmp/dist-test-task6x5P9o/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 18:39:45.832103 24012 mini_ranger.cc:193] Using host: 127.23.115.20
I20260620 18:39:45.846493 24012 mini_ranger.cc:227] Ranger admin URL: http://127.23.115.20:40423
Jun 20, 2026 6:39:46 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 6:39:46 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 6:39:46 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 6:39:46 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 6:39:46 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 6:39:46 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 6:39:46 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 18:39:46.891559 24012 mini_ranger.cc:160] Time spent starting Ranger: real 8.779s user 0.010s sys 0.030s
Jun 20, 2026 6:40:08 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 6:40:09 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Jun 20, 2026 6:40:09 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 6:40:09 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 6:40:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 6:40:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 6:40:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 6:40:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 6:40:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 6:40:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 6:40:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 6:40:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 6:40:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 6:40:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 6:40:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 6:40:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 6:40:09 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 6:40:10 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 18:40:10.683862 24012 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.23.115.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.23.115.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.23.115.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/krb5kdc/rangerkms_127.23.115.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.23.115.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/krb5kdc/rangerkms_127.23.115.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.23.115.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.23.115.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.115.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/krb5kdc/HTTP_127.23.115.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.23.115.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/krb5kdc/HTTP_127.23.115.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 18:40:10 dist-test-slave-d645 krb5kdc[2397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980810, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 18:40:10.867995 24012 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 18:40:10.891285 24012 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 18:40:11.932722 24012 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 18:40:11.932884 24012 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/minicluster-data/ranger-kms
I20260620 18:40:11.933012 24012 mini_ranger_kms.cc:227] Using postgres at 127.23.115.20:59515
2026-06-20 18:40:12,368 [I] DB FLAVOR :POSTGRES
2026-06-20 18:40:12,368 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 18:40:12,368 [I] Checking connection
2026-06-20 18:40:12,368 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6x5P9o/build/asan/bin/postgresql.jar:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.115.20:59515/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 18:40:12,621 [I] connection success
2026-06-20 18:40:12,621 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 18:40:12,621 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 18:40:12,621 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6x5P9o/build/asan/bin/postgresql.jar:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.115.20:59515/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 18:40:12,885 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 18:40:12,886 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 18:40:12,886 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 18:40:12,886 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6x5P9o/build/asan/bin/postgresql.jar:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.115.20:59515/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 18:40:13,166 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 18:40:13.644181 24012 mini_ranger_kms.cc:310] Created kms service
I20260620 18:40:13.852712 24012 mini_ranger_kms.cc:326] Created kudu user
I20260620 18:40:13.897662 24012 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 18:40:13 dist-test-slave-d645 krb5kdc[2397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980810, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.23.115.20@KRBTEST.COM
I20260620 18:40:14.351004 24012 mini_ranger_kms.cc:384] Added ranger policy
I20260620 18:40:14.351316 24012 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/minicluster-data/ranger-kms:/tmp/dist-test-task6x5P9o/build/asan/bin/postgresql.jar:/tmp/dist-test-task6x5P9o/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task6x5P9o/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 18:40:14.351361 24012 mini_ranger_kms.cc:240] Using host: 127.23.115.20
I20260620 18:40:14.376597 24012 mini_ranger_kms.cc:276] Ranger KMS PID: 2791
I20260620 18:40:14.376785 24012 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.23.115.20:36115
Jun 20, 2026 6:40:15 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 6:40:15 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 6:40:15 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 6:40:15 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 6:40:15 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 6:40:15 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Jun 20 18:40:16 dist-test-slave-d645 krb5kdc[2397](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781980816, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.23.115.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:40:16.515677 24012 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.648s user 0.132s sys 0.330s
I20260620 18:40:16.515942 24012 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 18:40:16.516156 24012 mini_ranger_kms.cc:401] 127.23.115.20:36115/kms/v1/keys
Jun 20 18:40:16 dist-test-slave-d645 krb5kdc[2397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980816, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.23.115.20@KRBTEST.COM for HTTP/127.23.115.20@KRBTEST.COM
Jun 20 18:40:17 dist-test-slave-d645 krb5kdc[2397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980816, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.23.115.20@KRBTEST.COM for HTTP/127.23.115.20@KRBTEST.COM
Jun 20 18:40:19 dist-test-slave-d645 krb5kdc[2397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980819, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.23.115.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.115.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.115.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab.
Entry for principal kudu/127.23.115.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab.
I20260620 18:40:19.728983 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.115.62:35261
--webserver_interface=127.23.115.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab
--principal=kudu/127.23.115.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:34763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.115.62:35261
--ranger_config_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.115.20:36115/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:40:20.070916 2841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:40:20.071280 2841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:40:20.071420 2841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:40:20.078734 2841 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:40:20.078868 2841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:40:20.078965 2841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:40:20.079051 2841 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:40:20.079118 2841 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:40:20.089390 2841 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:34763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/minicluster-data/master-0/wal
--ranger_kms_url=127.23.115.20:36115/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.23.115.62:35261
--ranger_config_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab
--principal=kudu/127.23.115.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.115.62:35261
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.115.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:40:20.090668 2841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:40:20.091952 2841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:40:20.105338 2847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:40:20.105433 2846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:40:20.105234 2849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:40:20.105409 2841 server_base.cc:1034] running on GCE node
I20260620 18:40:20.106644 2841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:40:20.108246 2841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:40:20.109561 2841 hybrid_clock.cc:648] HybridClock initialized: now 1781980820109489 us; error 61 us; skew 500 ppm
I20260620 18:40:20.109930 2841 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:40:20 dist-test-slave-d645 krb5kdc[2397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980820, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:40:20.117179 2841 init.cc:377] Logged in from keytab as kudu/127.23.115.62@KRBTEST.COM (short username kudu)
I20260620 18:40:20.118738 2841 webserver.cc:460] Webserver started at http://127.23.115.62:37967/ using document root <none> and password file <none>
I20260620 18:40:20.119429 2841 fs_manager.cc:359] Metadata directory not provided
I20260620 18:40:20.119544 2841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:40:20.119825 2841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 18:40:20 dist-test-slave-d645 krb5kdc[2397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980820, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.62@KRBTEST.COM for HTTP/127.23.115.20@KRBTEST.COM
I20260620 18:40:20.181423 2841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/minicluster-data/master-0/data/instance:
uuid: "c5c4212f13c34ea6b649d74194dde649"
format_stamp: "Formatted at 2026-06-20 18:40:20 on dist-test-slave-d645"
server_key: "2f4bafed15213a508c05c8c9285846cd"
server_key_iv: "015ecf329402f017ff0a60f6a4ad0b06"
server_key_version: "kuduclusterkey@0"
I20260620 18:40:20.182238 2841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/minicluster-data/master-0/wal/instance:
uuid: "c5c4212f13c34ea6b649d74194dde649"
format_stamp: "Formatted at 2026-06-20 18:40:20 on dist-test-slave-d645"
server_key: "2f4bafed15213a508c05c8c9285846cd"
server_key_iv: "015ecf329402f017ff0a60f6a4ad0b06"
server_key_version: "kuduclusterkey@0"
I20260620 18:40:20.187493 2841 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 18:40:20.206192 2856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:40:20.207834 2841 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:40:20.208019 2841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/minicluster-data/master-0/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/minicluster-data/master-0/wal
uuid: "c5c4212f13c34ea6b649d74194dde649"
format_stamp: "Formatted at 2026-06-20 18:40:20 on dist-test-slave-d645"
server_key: "2f4bafed15213a508c05c8c9285846cd"
server_key_iv: "015ecf329402f017ff0a60f6a4ad0b06"
server_key_version: "kuduclusterkey@0"
I20260620 18:40:20.208228 2841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:40:20.221679 2841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:40:20.226609 2841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:40:20.227072 2841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:40:20.248189 2841 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.62:35261
I20260620 18:40:20.248282 2907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.62:35261 every 8 connection(s)
I20260620 18:40:20.249742 2841 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
I20260620 18:40:20.250458 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 2841
I20260620 18:40:20.250650 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/minicluster-data/master-0/wal/instance
I20260620 18:40:20.251690 24012 external_mini_cluster.cc:1428] Setting key 056185c73f0b107aa62fe2e302726ce7
I20260620 18:40:20.255543 2908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 18:40:20 dist-test-slave-d645 krb5kdc[2397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980819, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.115.62@KRBTEST.COM
I20260620 18:40:20.268755 2908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5c4212f13c34ea6b649d74194dde649: Bootstrap starting.
I20260620 18:40:20.272740 2908 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c5c4212f13c34ea6b649d74194dde649: Neither blocks nor log segments found. Creating new log.
I20260620 18:40:20.273804 2908 log.cc:824] T 00000000000000000000000000000000 P c5c4212f13c34ea6b649d74194dde649: Log is configured to *not* fsync() on all Append() calls
I20260620 18:40:20.277187 2908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5c4212f13c34ea6b649d74194dde649: No bootstrap required, opened a new log
I20260620 18:40:20.278018 2911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:20.253891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35338 (local address 127.23.115.62:35261)
0620 18:40:20.254764 (+ 873us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:20.254777 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:20.254821 (+ 44us) server_negotiation.cc:407] Connection header received
0620 18:40:20.256738 (+ 1917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:20.256762 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:20.257158 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:20.257528 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:40:20.259569 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:20.261431 (+ 1862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:20.262788 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:20.263609 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:20.263796 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:20.267919 (+ 4123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:40:20.267931 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:40:20.267952 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:40:20.268002 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:40:20.271811 (+ 3809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:40:20.273101 (+ 1290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:40:20.273113 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:40:20.273123 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:40:20.273409 (+ 286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:40:20.273918 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:40:20.273923 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:40:20.273925 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:40:20.274684 (+ 759us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:40:20.275479 (+ 795us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:20.276051 (+ 572us) server_negotiation.cc:299] Negotiation successful
0620 18:40:20.276614 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":649,"thread_start_us":431,"threads_started":1}
I20260620 18:40:20.285616 2908 raft_consensus.cc:348] T 00000000000000000000000000000000 P c5c4212f13c34ea6b649d74194dde649 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5c4212f13c34ea6b649d74194dde649" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 35261 } }
I20260620 18:40:20.286010 2908 raft_consensus.cc:374] T 00000000000000000000000000000000 P c5c4212f13c34ea6b649d74194dde649 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:40:20.286105 2908 raft_consensus.cc:729] T 00000000000000000000000000000000 P c5c4212f13c34ea6b649d74194dde649 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5c4212f13c34ea6b649d74194dde649, State: Initialized, Role: FOLLOWER
I20260620 18:40:20.286667 2908 consensus_queue.cc:260] T 00000000000000000000000000000000 P c5c4212f13c34ea6b649d74194dde649 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5c4212f13c34ea6b649d74194dde649" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 35261 } }
I20260620 18:40:20.286890 2908 raft_consensus.cc:388] T 00000000000000000000000000000000 P c5c4212f13c34ea6b649d74194dde649 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:40:20.287055 2908 raft_consensus.cc:482] T 00000000000000000000000000000000 P c5c4212f13c34ea6b649d74194dde649 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:40:20.287204 2908 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c5c4212f13c34ea6b649d74194dde649 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:40:20.289662 2908 raft_consensus.cc:504] T 00000000000000000000000000000000 P c5c4212f13c34ea6b649d74194dde649 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5c4212f13c34ea6b649d74194dde649" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 35261 } }
I20260620 18:40:20.290135 2908 leader_election.cc:302] T 00000000000000000000000000000000 P c5c4212f13c34ea6b649d74194dde649 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c5c4212f13c34ea6b649d74194dde649; no voters:
I20260620 18:40:20.290732 2908 leader_election.cc:288] T 00000000000000000000000000000000 P c5c4212f13c34ea6b649d74194dde649 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:40:20.290853 2913 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c5c4212f13c34ea6b649d74194dde649 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:40:20.291486 2913 raft_consensus.cc:686] T 00000000000000000000000000000000 P c5c4212f13c34ea6b649d74194dde649 [term 1 LEADER]: Becoming Leader. State: Replica: c5c4212f13c34ea6b649d74194dde649, State: Running, Role: LEADER
I20260620 18:40:20.292099 2913 consensus_queue.cc:237] T 00000000000000000000000000000000 P c5c4212f13c34ea6b649d74194dde649 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5c4212f13c34ea6b649d74194dde649" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 35261 } }
I20260620 18:40:20.292865 2908 sys_catalog.cc:564] T 00000000000000000000000000000000 P c5c4212f13c34ea6b649d74194dde649 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:40:20.295254 2915 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5c4212f13c34ea6b649d74194dde649 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c5c4212f13c34ea6b649d74194dde649. Latest consensus state: current_term: 1 leader_uuid: "c5c4212f13c34ea6b649d74194dde649" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5c4212f13c34ea6b649d74194dde649" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 35261 } } }
I20260620 18:40:20.295668 2915 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5c4212f13c34ea6b649d74194dde649 [sys.catalog]: This master's current role is: LEADER
I20260620 18:40:20.295516 2914 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5c4212f13c34ea6b649d74194dde649 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c5c4212f13c34ea6b649d74194dde649" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5c4212f13c34ea6b649d74194dde649" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 35261 } } }
I20260620 18:40:20.295789 2914 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5c4212f13c34ea6b649d74194dde649 [sys.catalog]: This master's current role is: LEADER
I20260620 18:40:20.299750 2922 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:40:20.300429 2908 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 18:40:20.311514 2922 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:40:20.333331 2922 catalog_manager.cc:1347] Generated new cluster ID: 724fd2034baf42479376db2462fbf91b
I20260620 18:40:20.333467 2922 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:40:20.356629 2922 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:40:20.358647 2922 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:40:20.367985 2922 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c5c4212f13c34ea6b649d74194dde649: Generated new TSK 0
I20260620 18:40:20.369095 2922 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.115.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.115.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.115.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.1.keytab.
Entry for principal kudu/127.23.115.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.1.keytab.
I20260620 18:40:21.167728 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.115.20:36115/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.115.1:0
--local_ip_for_outbound_sockets=127.23.115.1
--webserver_interface=127.23.115.1
--webserver_port=0
--tserver_master_addrs=127.23.115.62:35261
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.1.keytab
--principal=kudu/127.23.115.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:34763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 18:40:21 dist-test-slave-d645 krb5kdc[2397](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781980821, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 18:40:21.627292 2953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:40:21.627769 2953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:40:21.627974 2953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:40:21.639874 2953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:40:21.640092 2953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:40:21.640434 2953 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.115.1
I20260620 18:40:21.656977 2953 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:34763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.23.115.20:36115/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.1.keytab
--principal=kudu/127.23.115.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.115.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.115.1
--webserver_port=0
--tserver_master_addrs=127.23.115.62:35261
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.115.1
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:40:21.658667 2953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:40:21.660498 2953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:40:21.678021 2964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:40:21.678185 2966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:40:21.678921 2963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:40:21.684988 2953 server_base.cc:1034] running on GCE node
I20260620 18:40:21.685940 2953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:40:21.687259 2953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:40:21.688553 2953 hybrid_clock.cc:648] HybridClock initialized: now 1781980821688455 us; error 85 us; skew 500 ppm
I20260620 18:40:21.688952 2953 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:40:21 dist-test-slave-d645 krb5kdc[2397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980821, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:40:21.694128 2953 init.cc:377] Logged in from keytab as kudu/127.23.115.1@KRBTEST.COM (short username kudu)
I20260620 18:40:21.696614 2953 webserver.cc:460] Webserver started at http://127.23.115.1:42371/ using document root <none> and password file <none>
I20260620 18:40:21.697711 2953 fs_manager.cc:359] Metadata directory not provided
I20260620 18:40:21.697880 2953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:40:21.698650 2953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 18:40:21 dist-test-slave-d645 krb5kdc[2397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980821, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.1@KRBTEST.COM for HTTP/127.23.115.20@KRBTEST.COM
I20260620 18:40:21.725090 2953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/minicluster-data/ts-0/data/instance:
uuid: "cde3a3270250472793136621fc039e6a"
format_stamp: "Formatted at 2026-06-20 18:40:21 on dist-test-slave-d645"
server_key: "bf2a7513d9864d64f1a29b8ff7c32e57"
server_key_iv: "314ffaa5c2e4025d341ad45628526c3a"
server_key_version: "kuduclusterkey@0"
I20260620 18:40:21.727454 2953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/minicluster-data/ts-0/wal/instance:
uuid: "cde3a3270250472793136621fc039e6a"
format_stamp: "Formatted at 2026-06-20 18:40:21 on dist-test-slave-d645"
server_key: "bf2a7513d9864d64f1a29b8ff7c32e57"
server_key_iv: "314ffaa5c2e4025d341ad45628526c3a"
server_key_version: "kuduclusterkey@0"
I20260620 18:40:21.737615 2953 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.006s sys 0.001s
I20260620 18:40:21.775676 2973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:40:21.777832 2953 fs_manager.cc:725] Time spent opening block manager: real 0.022s user 0.005s sys 0.012s
I20260620 18:40:21.778154 2953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/minicluster-data/ts-0/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/minicluster-data/ts-0/wal
uuid: "cde3a3270250472793136621fc039e6a"
format_stamp: "Formatted at 2026-06-20 18:40:21 on dist-test-slave-d645"
server_key: "bf2a7513d9864d64f1a29b8ff7c32e57"
server_key_iv: "314ffaa5c2e4025d341ad45628526c3a"
server_key_version: "kuduclusterkey@0"
I20260620 18:40:21.778542 2953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:40:21.833648 2953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:40:21.846261 2953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:40:21.846700 2953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:40:21.859285 2953 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:40:21.871240 2953 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:40:21.871455 2953 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:40:21.871677 2953 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:40:21.871801 2953 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 18:40:21 dist-test-slave-d645 krb5kdc[2397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980821, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.62@KRBTEST.COM for HTTP/127.23.115.20@KRBTEST.COM
Jun 20 18:40:22 dist-test-slave-d645 krb5kdc[2397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980821, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.62@KRBTEST.COM for HTTP/127.23.115.20@KRBTEST.COM
I20260620 18:40:22.345393 2953 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.1:32929
I20260620 18:40:22.345398 3086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.1:32929 every 8 connection(s)
I20260620 18:40:22.347813 2953 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/minicluster-data/ts-0/data/info.pb
I20260620 18:40:22.357364 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 2953
I20260620 18:40:22.357605 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/minicluster-data/ts-0/wal/instance
I20260620 18:40:22.358435 24012 external_mini_cluster.cc:1428] Setting key 95005f39f3ac674edb88b1a5dde9047d
Jun 20 18:40:22 dist-test-slave-d645 krb5kdc[2397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980821, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.1@KRBTEST.COM for kudu/127.23.115.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.115.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.115.2@KRBTEST.COM" created.
I20260620 18:40:22.417764 3089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:22.353105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:35261 (local address 127.23.115.1:56605)
0620 18:40:22.383763 (+ 30658us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:22.383852 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0620 18:40:22.391552 (+ 7700us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:40:22.394222 (+ 2670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:40:22.394238 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:40:22.394887 (+ 649us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:40:22.396016 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:22.396046 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:22.398066 (+ 2020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:22.398072 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:22.400469 (+ 2397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:22.400488 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:22.401902 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:22.401911 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:22.402467 (+ 556us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:22.403700 (+ 1233us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:40:22.403733 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:40:22.406671 (+ 2938us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:40:22.410785 (+ 4114us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:40:22.410794 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:40:22.410814 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:40:22.411425 (+ 611us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:40:22.411898 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:40:22.411903 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:40:22.411907 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:40:22.412174 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:40:22.414901 (+ 2727us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:40:22.414954 (+ 53us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:40:22.415457 (+ 503us) client_negotiation.cc:769] Sending connection context
0620 18:40:22.415772 (+ 315us) client_negotiation.cc:240] Negotiation successful
0620 18:40:22.416333 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29918,"thread_start_us":33379,"threads_started":1}
I20260620 18:40:22.423349 3090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:22.353834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.1:56605 (local address 127.23.115.62:35261)
0620 18:40:22.382549 (+ 28715us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:22.382558 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:22.385165 (+ 2607us) server_negotiation.cc:407] Connection header received
0620 18:40:22.393277 (+ 8112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:22.393284 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:22.393454 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:22.393683 (+ 229us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:40:22.396323 (+ 2640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:22.397850 (+ 1527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:22.400676 (+ 2826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:22.401308 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:22.401434 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:22.407004 (+ 5570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:40:22.407012 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:40:22.407017 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:40:22.407062 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:40:22.410576 (+ 3514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:40:22.411592 (+ 1016us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:40:22.411598 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:40:22.411602 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:40:22.411774 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:40:22.414351 (+ 2577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:40:22.414356 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:40:22.414360 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:40:22.414739 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:40:22.414896 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:22.422688 (+ 7792us) server_negotiation.cc:299] Negotiation successful
0620 18:40:22.422977 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28517,"thread_start_us":28721,"threads_started":1}
I20260620 18:40:22.425073 3087 heartbeater.cc:347] Connected to a master server at 127.23.115.62:35261
I20260620 18:40:22.425417 3087 heartbeater.cc:464] Registering TS with master...
I20260620 18:40:22.426126 3087 heartbeater.cc:511] Master 127.23.115.62:35261 requested a full tablet report, sending...
I20260620 18:40:22.428510 2873 ts_manager.cc:194] Registered new tserver with Master: cde3a3270250472793136621fc039e6a (127.23.115.1:32929)
I20260620 18:40:22.430459 2873 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.115.1@KRBTEST.COM'} at 127.23.115.1:56605
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.115.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.2.keytab.
Entry for principal kudu/127.23.115.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.2.keytab.
W20260620 18:40:22.474468 24012 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.23.115.2: real 0.112s user 0.003s sys 0.013s
I20260620 18:40:22.475651 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.115.20:36115/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.115.2:0
--local_ip_for_outbound_sockets=127.23.115.2
--webserver_interface=127.23.115.2
--webserver_port=0
--tserver_master_addrs=127.23.115.62:35261
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.2.keytab
--principal=kudu/127.23.115.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:34763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:40:22.912472 3094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:40:22.912818 3094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:40:22.912966 3094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:40:22.921134 3094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:40:22.921301 3094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:40:22.921451 3094 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.115.2
I20260620 18:40:22.932444 3094 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:34763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.23.115.20:36115/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.2.keytab
--principal=kudu/127.23.115.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.115.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.115.2
--webserver_port=0
--tserver_master_addrs=127.23.115.62:35261
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.115.2
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:40:22.933786 3094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:40:22.935118 3094 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:40:22.948504 3106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:40:22.948449 3107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:40:22.949159 3109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:40:22.949332 3094 server_base.cc:1034] running on GCE node
I20260620 18:40:22.949970 3094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:40:22.950954 3094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:40:22.952198 3094 hybrid_clock.cc:648] HybridClock initialized: now 1781980822952148 us; error 41 us; skew 500 ppm
I20260620 18:40:22.952491 3094 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:40:22 dist-test-slave-d645 krb5kdc[2397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980822, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:40:22.955497 3094 init.cc:377] Logged in from keytab as kudu/127.23.115.2@KRBTEST.COM (short username kudu)
I20260620 18:40:22.957036 3094 webserver.cc:460] Webserver started at http://127.23.115.2:44205/ using document root <none> and password file <none>
I20260620 18:40:22.957830 3094 fs_manager.cc:359] Metadata directory not provided
I20260620 18:40:22.957973 3094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:40:22.958277 3094 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 18:40:22 dist-test-slave-d645 krb5kdc[2397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980822, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.2@KRBTEST.COM for HTTP/127.23.115.20@KRBTEST.COM
I20260620 18:40:22.976457 3094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/minicluster-data/ts-1/data/instance:
uuid: "ec14a1049305490fa2a9b11d750ce8c1"
format_stamp: "Formatted at 2026-06-20 18:40:22 on dist-test-slave-d645"
server_key: "2e1248cd1dc164aaeec64fbc9e938719"
server_key_iv: "621fdd2ae630e2c92919adb3af67becb"
server_key_version: "kuduclusterkey@0"
I20260620 18:40:22.977605 3094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/minicluster-data/ts-1/wal/instance:
uuid: "ec14a1049305490fa2a9b11d750ce8c1"
format_stamp: "Formatted at 2026-06-20 18:40:22 on dist-test-slave-d645"
server_key: "2e1248cd1dc164aaeec64fbc9e938719"
server_key_iv: "621fdd2ae630e2c92919adb3af67becb"
server_key_version: "kuduclusterkey@0"
I20260620 18:40:22.992686 3094 fs_manager.cc:691] Time spent creating directory manager: real 0.014s user 0.013s sys 0.000s
I20260620 18:40:23.013273 3116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:40:23.015236 3094 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260620 18:40:23.015429 3094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/minicluster-data/ts-1/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/minicluster-data/ts-1/wal
uuid: "ec14a1049305490fa2a9b11d750ce8c1"
format_stamp: "Formatted at 2026-06-20 18:40:22 on dist-test-slave-d645"
server_key: "2e1248cd1dc164aaeec64fbc9e938719"
server_key_iv: "621fdd2ae630e2c92919adb3af67becb"
server_key_version: "kuduclusterkey@0"
I20260620 18:40:23.015693 3094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:40:23.041190 3094 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:40:23.046099 3094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:40:23.046422 3094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:40:23.047602 3094 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:40:23.049239 3094 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:40:23.049338 3094 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:40:23.049463 3094 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:40:23.049525 3094 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:40:23.083523 3094 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.2:37595
I20260620 18:40:23.083539 3228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.2:37595 every 8 connection(s)
I20260620 18:40:23.085222 3094 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/minicluster-data/ts-1/data/info.pb
I20260620 18:40:23.087206 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 3094
I20260620 18:40:23.087376 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/minicluster-data/ts-1/wal/instance
I20260620 18:40:23.087962 24012 external_mini_cluster.cc:1428] Setting key 043862e737eb4e80c4ec6596b4b9ad33
Jun 20 18:40:23 dist-test-slave-d645 krb5kdc[2397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980822, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.2@KRBTEST.COM for kudu/127.23.115.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.115.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.115.3@KRBTEST.COM" created.
I20260620 18:40:23.127065 3231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:23.089111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.2:32795 (local address 127.23.115.62:35261)
0620 18:40:23.101750 (+ 12639us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:23.101756 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:23.101780 (+ 24us) server_negotiation.cc:407] Connection header received
0620 18:40:23.107073 (+ 5293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:23.107080 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:23.107178 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:23.107290 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:40:23.109682 (+ 2392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:23.110766 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:23.113434 (+ 2668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:23.113971 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:23.114101 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:23.118833 (+ 4732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:40:23.118841 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:40:23.118846 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:40:23.118901 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:40:23.122595 (+ 3694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:40:23.123669 (+ 1074us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:40:23.123675 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:40:23.123680 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:40:23.123864 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:40:23.124642 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:40:23.124648 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:40:23.124652 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:40:23.125143 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:40:23.125332 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:23.126230 (+ 898us) server_negotiation.cc:299] Negotiation successful
0620 18:40:23.126550 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12514,"thread_start_us":12664,"threads_started":1}
I20260620 18:40:23.127987 3232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:23.089899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:35261 (local address 127.23.115.2:32795)
0620 18:40:23.101298 (+ 11399us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:23.101359 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 18:40:23.103588 (+ 2229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:40:23.107630 (+ 4042us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:40:23.107644 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:40:23.108431 (+ 787us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:40:23.109383 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:23.109415 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:23.110980 (+ 1565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:23.110987 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:23.113262 (+ 2275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:23.113280 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:23.114238 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:23.114248 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:23.114669 (+ 421us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:23.115556 (+ 887us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:40:23.115592 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:40:23.118544 (+ 2952us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:40:23.122760 (+ 4216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:40:23.122768 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:40:23.122783 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:40:23.123511 (+ 728us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:40:23.124248 (+ 737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:40:23.124252 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:40:23.124255 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:40:23.124503 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:40:23.125291 (+ 788us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:40:23.125299 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:40:23.125791 (+ 492us) client_negotiation.cc:769] Sending connection context
0620 18:40:23.126069 (+ 278us) client_negotiation.cc:240] Negotiation successful
0620 18:40:23.126627 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11027,"thread_start_us":12946,"threads_started":1}
I20260620 18:40:23.129513 3229 heartbeater.cc:347] Connected to a master server at 127.23.115.62:35261
I20260620 18:40:23.129887 3229 heartbeater.cc:464] Registering TS with master...
I20260620 18:40:23.130815 3229 heartbeater.cc:511] Master 127.23.115.62:35261 requested a full tablet report, sending...
I20260620 18:40:23.132956 2873 ts_manager.cc:194] Registered new tserver with Master: ec14a1049305490fa2a9b11d750ce8c1 (127.23.115.2:37595)
I20260620 18:40:23.133932 2873 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.115.2@KRBTEST.COM'} at 127.23.115.2:32795
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.115.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.3.keytab.
Entry for principal kudu/127.23.115.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.3.keytab.
I20260620 18:40:23.154376 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.115.20:36115/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.115.3:0
--local_ip_for_outbound_sockets=127.23.115.3
--webserver_interface=127.23.115.3
--webserver_port=0
--tserver_master_addrs=127.23.115.62:35261
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.3.keytab
--principal=kudu/127.23.115.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:34763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:40:23.434413 3087 heartbeater.cc:503] Master 127.23.115.62:35261 was elected leader, sending a full tablet report...
W20260620 18:40:23.495368 3235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:40:23.495674 3235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:40:23.495790 3235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:40:23.503121 3235 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:40:23.503261 3235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:40:23.503420 3235 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.115.3
I20260620 18:40:23.514076 3235 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:34763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.23.115.20:36115/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.3.keytab
--principal=kudu/127.23.115.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.115.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.115.3
--webserver_port=0
--tserver_master_addrs=127.23.115.62:35261
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.115.3
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:40:23.515419 3235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:40:23.516743 3235 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:40:23.530208 3240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:40:23.530180 3241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:40:23.530315 3243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:40:23.530918 3235 server_base.cc:1034] running on GCE node
I20260620 18:40:23.531561 3235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:40:23.532507 3235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:40:23.533826 3235 hybrid_clock.cc:648] HybridClock initialized: now 1781980823533731 us; error 37 us; skew 500 ppm
I20260620 18:40:23.534380 3235 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:40:23 dist-test-slave-d645 krb5kdc[2397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980823, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:40:23.537364 3235 init.cc:377] Logged in from keytab as kudu/127.23.115.3@KRBTEST.COM (short username kudu)
I20260620 18:40:23.538763 3235 webserver.cc:460] Webserver started at http://127.23.115.3:41627/ using document root <none> and password file <none>
I20260620 18:40:23.539500 3235 fs_manager.cc:359] Metadata directory not provided
I20260620 18:40:23.539651 3235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:40:23.539947 3235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 18:40:23 dist-test-slave-d645 krb5kdc[2397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980823, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.3@KRBTEST.COM for HTTP/127.23.115.20@KRBTEST.COM
I20260620 18:40:23.554759 3235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/minicluster-data/ts-2/data/instance:
uuid: "193a123616694962b2cd4044f703bb83"
format_stamp: "Formatted at 2026-06-20 18:40:23 on dist-test-slave-d645"
server_key: "a957978311453e180d69986503ee61ad"
server_key_iv: "68991d0513eff2dac880af07c4a15419"
server_key_version: "kuduclusterkey@0"
I20260620 18:40:23.555689 3235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/minicluster-data/ts-2/wal/instance:
uuid: "193a123616694962b2cd4044f703bb83"
format_stamp: "Formatted at 2026-06-20 18:40:23 on dist-test-slave-d645"
server_key: "a957978311453e180d69986503ee61ad"
server_key_iv: "68991d0513eff2dac880af07c4a15419"
server_key_version: "kuduclusterkey@0"
I20260620 18:40:23.561264 3235 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:40:23.578303 3250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:40:23.580083 3235 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260620 18:40:23.580263 3235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/minicluster-data/ts-2/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/minicluster-data/ts-2/wal
uuid: "193a123616694962b2cd4044f703bb83"
format_stamp: "Formatted at 2026-06-20 18:40:23 on dist-test-slave-d645"
server_key: "a957978311453e180d69986503ee61ad"
server_key_iv: "68991d0513eff2dac880af07c4a15419"
server_key_version: "kuduclusterkey@0"
I20260620 18:40:23.580544 3235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:40:23.633352 3235 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:40:23.638411 3235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:40:23.638744 3235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:40:23.639878 3235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:40:23.641429 3235 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:40:23.641520 3235 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:40:23.641631 3235 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:40:23.641686 3235 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:40:23.674211 3235 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.3:33547
I20260620 18:40:23.674239 3362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.3:33547 every 8 connection(s)
I20260620 18:40:23.675851 3235 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/minicluster-data/ts-2/data/info.pb
I20260620 18:40:23.678709 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 3235
I20260620 18:40:23.678894 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/minicluster-data/ts-2/wal/instance
I20260620 18:40:23.679497 24012 external_mini_cluster.cc:1428] Setting key 837dbda93b6f14322743b24f29c44b87
Jun 20 18:40:23 dist-test-slave-d645 krb5kdc[2397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980823, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.3@KRBTEST.COM for kudu/127.23.115.62@KRBTEST.COM
I20260620 18:40:23.702004 3365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:23.678948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.3:47405 (local address 127.23.115.62:35261)
0620 18:40:23.687303 (+ 8355us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:23.687307 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:23.687322 (+ 15us) server_negotiation.cc:407] Connection header received
0620 18:40:23.687372 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:23.687376 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:23.687457 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:23.687548 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:40:23.689019 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:23.689745 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:23.691304 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:23.691734 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:23.691846 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:23.695765 (+ 3919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:40:23.695770 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:40:23.695774 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:40:23.695805 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:40:23.698481 (+ 2676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:40:23.699408 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:40:23.699412 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:40:23.699414 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:40:23.699531 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:40:23.700032 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:40:23.700036 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:40:23.700038 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:40:23.700356 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:40:23.700474 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:23.701294 (+ 820us) server_negotiation.cc:299] Negotiation successful
0620 18:40:23.701516 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8250,"thread_start_us":8129,"threads_started":1}
I20260620 18:40:23.703130 3366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:23.679539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:35261 (local address 127.23.115.3:47405)
0620 18:40:23.683680 (+ 4141us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:23.683749 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0620 18:40:23.686328 (+ 2579us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:40:23.687808 (+ 1480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:40:23.687817 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:40:23.688261 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:40:23.688841 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:23.688857 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:23.689902 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:23.689905 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:23.691168 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:23.691180 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:23.691977 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:23.691985 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:23.692194 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:23.692928 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:40:23.692951 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:40:23.695547 (+ 2596us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:40:23.698624 (+ 3077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:40:23.698634 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:40:23.698652 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:40:23.699260 (+ 608us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:40:23.699624 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:40:23.699629 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:40:23.699632 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:40:23.699884 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:40:23.700480 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:40:23.700488 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:40:23.700900 (+ 412us) client_negotiation.cc:769] Sending connection context
0620 18:40:23.701192 (+ 292us) client_negotiation.cc:240] Negotiation successful
0620 18:40:23.701714 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3806,"thread_start_us":3570,"threads_started":1}
I20260620 18:40:23.704806 3363 heartbeater.cc:347] Connected to a master server at 127.23.115.62:35261
I20260620 18:40:23.705083 3363 heartbeater.cc:464] Registering TS with master...
I20260620 18:40:23.705847 3363 heartbeater.cc:511] Master 127.23.115.62:35261 requested a full tablet report, sending...
I20260620 18:40:23.707502 2873 ts_manager.cc:194] Registered new tserver with Master: 193a123616694962b2cd4044f703bb83 (127.23.115.3:33547)
I20260620 18:40:23.708029 24012 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:40:23.708626 2873 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.115.3@KRBTEST.COM'} at 127.23.115.3:47405
I20260620 18:40:23.726191 3365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:23.713457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35344 (local address 127.23.115.62:35261)
0620 18:40:23.713615 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:23.713620 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:23.714116 (+ 496us) server_negotiation.cc:407] Connection header received
0620 18:40:23.714273 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:23.714279 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:23.714353 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:23.714485 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:40:23.715904 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:23.716613 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:23.717949 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:23.718379 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:23.718506 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:23.720838 (+ 2332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:40:23.720842 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:40:23.720846 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:40:23.720877 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:40:23.723570 (+ 2693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:40:23.724352 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:40:23.724356 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:40:23.724359 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:40:23.724479 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:40:23.724904 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:40:23.724909 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:40:23.724911 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:40:23.725185 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:40:23.725397 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:23.725749 (+ 352us) server_negotiation.cc:299] Negotiation successful
0620 18:40:23.725925 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 18:40:23.731206 2873 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35344:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:40:23.741456 2873 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:40:23.775372 3376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:23.754825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.2:37595 (local address 127.0.0.1:46342)
0620 18:40:23.764073 (+ 9248us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:23.766074 (+ 2001us) client_negotiation.cc:174] Beginning negotiation
0620 18:40:23.766262 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:40:23.767370 (+ 1108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:40:23.767377 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:40:23.767404 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:40:23.767821 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:23.767836 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:23.769562 (+ 1726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:23.769568 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:23.772155 (+ 2587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:23.772174 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:23.774311 (+ 2137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:23.774317 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:23.774504 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:23.774550 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 18:40:23.774676 (+ 126us) client_negotiation.cc:240] Negotiation successful
0620 18:40:23.774888 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8911,"mutex_wait_us":136,"thread_start_us":8790,"threads_started":1}
I20260620 18:40:23.775969 3379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:23.755221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41692 (local address 127.23.115.3:33547)
0620 18:40:23.763692 (+ 8471us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:23.763706 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:23.765668 (+ 1962us) server_negotiation.cc:407] Connection header received
0620 18:40:23.767217 (+ 1549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:23.767225 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:23.767446 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:23.767662 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:40:23.768743 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:23.770231 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:23.772940 (+ 2709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:23.774303 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:23.774418 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:23.774611 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:23.775476 (+ 865us) server_negotiation.cc:299] Negotiation successful
0620 18:40:23.775647 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8064,"thread_start_us":7885,"threads_started":1}
I20260620 18:40:23.776063 3378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:23.755330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46342 (local address 127.23.115.2:37595)
0620 18:40:23.764393 (+ 9063us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:23.764406 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:23.766452 (+ 2046us) server_negotiation.cc:407] Connection header received
0620 18:40:23.766564 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:23.766573 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:23.766889 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:23.767104 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:40:23.768031 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:23.769410 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:23.772336 (+ 2926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:23.773862 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:23.773962 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:23.774162 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:23.775615 (+ 1453us) server_negotiation.cc:299] Negotiation successful
0620 18:40:23.775779 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8882,"thread_start_us":9230,"threads_started":1}
I20260620 18:40:23.776535 3377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:23.754852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.3:33547 (local address 127.0.0.1:41692)
0620 18:40:23.765517 (+ 10665us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:23.765557 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 18:40:23.767045 (+ 1488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:40:23.767641 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:40:23.767648 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:40:23.767670 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:40:23.768580 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:23.768593 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:23.770386 (+ 1793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:23.770391 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:23.772713 (+ 2322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:23.772738 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:23.774747 (+ 2009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:23.774753 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:23.774954 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:23.774986 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 18:40:23.775069 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 18:40:23.775187 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10554,"mutex_wait_us":28,"thread_start_us":10334,"threads_started":1}
I20260620 18:40:23.777051 3380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:23.765590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.1:32929 (local address 127.0.0.1:59232)
0620 18:40:23.766132 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:23.766149 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:40:23.766261 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:40:23.768075 (+ 1814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:40:23.768080 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:40:23.768102 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:40:23.769126 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:23.769138 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:23.771037 (+ 1899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:23.771043 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:23.774094 (+ 3051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:23.774107 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:23.776055 (+ 1948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:23.776062 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:23.776218 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:23.776248 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 18:40:23.776324 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 18:40:23.776450 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"mutex_wait_us":628,"thread_start_us":305,"threads_started":1}
I20260620 18:40:23.779733 3381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:23.765939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59232 (local address 127.23.115.1:32929)
0620 18:40:23.767571 (+ 1632us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:23.767579 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:23.767605 (+ 26us) server_negotiation.cc:407] Connection header received
0620 18:40:23.767692 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:23.767698 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:23.767892 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:23.768161 (+ 269us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:40:23.769315 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:23.770843 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:23.774264 (+ 3421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:23.775841 (+ 1577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:23.776640 (+ 799us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:23.779112 (+ 2472us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:23.779213 (+ 101us) server_negotiation.cc:299] Negotiation successful
0620 18:40:23.779399 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1467,"thread_start_us":1314,"threads_started":1}
I20260620 18:40:23.781394 3164 tablet_service.cc:1467] Processing CreateTablet for tablet c2897d812a2e4ad79b0c91448a4aeb06 (DEFAULT_TABLE table=test-table [id=0df382dc85c84e1093b9502988a8cb00]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:40:23.781667 3298 tablet_service.cc:1467] Processing CreateTablet for tablet c2897d812a2e4ad79b0c91448a4aeb06 (DEFAULT_TABLE table=test-table [id=0df382dc85c84e1093b9502988a8cb00]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:40:23.782892 3164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2897d812a2e4ad79b0c91448a4aeb06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:40:23.783097 3298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2897d812a2e4ad79b0c91448a4aeb06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:40:23.783488 3021 tablet_service.cc:1467] Processing CreateTablet for tablet c2897d812a2e4ad79b0c91448a4aeb06 (DEFAULT_TABLE table=test-table [id=0df382dc85c84e1093b9502988a8cb00]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:40:23.784823 3021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2897d812a2e4ad79b0c91448a4aeb06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:40:23.802798 3384 tablet_bootstrap.cc:492] T c2897d812a2e4ad79b0c91448a4aeb06 P 193a123616694962b2cd4044f703bb83: Bootstrap starting.
I20260620 18:40:23.809131 3383 tablet_bootstrap.cc:492] T c2897d812a2e4ad79b0c91448a4aeb06 P cde3a3270250472793136621fc039e6a: Bootstrap starting.
I20260620 18:40:23.809990 3382 tablet_bootstrap.cc:492] T c2897d812a2e4ad79b0c91448a4aeb06 P ec14a1049305490fa2a9b11d750ce8c1: Bootstrap starting.
I20260620 18:40:23.811794 3384 tablet_bootstrap.cc:654] T c2897d812a2e4ad79b0c91448a4aeb06 P 193a123616694962b2cd4044f703bb83: Neither blocks nor log segments found. Creating new log.
I20260620 18:40:23.813347 3384 log.cc:824] T c2897d812a2e4ad79b0c91448a4aeb06 P 193a123616694962b2cd4044f703bb83: Log is configured to *not* fsync() on all Append() calls
I20260620 18:40:23.814242 3382 tablet_bootstrap.cc:654] T c2897d812a2e4ad79b0c91448a4aeb06 P ec14a1049305490fa2a9b11d750ce8c1: Neither blocks nor log segments found. Creating new log.
I20260620 18:40:23.814270 3383 tablet_bootstrap.cc:654] T c2897d812a2e4ad79b0c91448a4aeb06 P cde3a3270250472793136621fc039e6a: Neither blocks nor log segments found. Creating new log.
I20260620 18:40:23.815676 3382 log.cc:824] T c2897d812a2e4ad79b0c91448a4aeb06 P ec14a1049305490fa2a9b11d750ce8c1: Log is configured to *not* fsync() on all Append() calls
I20260620 18:40:23.815676 3383 log.cc:824] T c2897d812a2e4ad79b0c91448a4aeb06 P cde3a3270250472793136621fc039e6a: Log is configured to *not* fsync() on all Append() calls
I20260620 18:40:23.825229 3382 tablet_bootstrap.cc:492] T c2897d812a2e4ad79b0c91448a4aeb06 P ec14a1049305490fa2a9b11d750ce8c1: No bootstrap required, opened a new log
I20260620 18:40:23.825520 3382 ts_tablet_manager.cc:1397] T c2897d812a2e4ad79b0c91448a4aeb06 P ec14a1049305490fa2a9b11d750ce8c1: Time spent bootstrapping tablet: real 0.016s user 0.010s sys 0.000s
I20260620 18:40:23.825547 3384 tablet_bootstrap.cc:492] T c2897d812a2e4ad79b0c91448a4aeb06 P 193a123616694962b2cd4044f703bb83: No bootstrap required, opened a new log
I20260620 18:40:23.825842 3384 ts_tablet_manager.cc:1397] T c2897d812a2e4ad79b0c91448a4aeb06 P 193a123616694962b2cd4044f703bb83: Time spent bootstrapping tablet: real 0.023s user 0.006s sys 0.010s
I20260620 18:40:23.825991 3383 tablet_bootstrap.cc:492] T c2897d812a2e4ad79b0c91448a4aeb06 P cde3a3270250472793136621fc039e6a: No bootstrap required, opened a new log
I20260620 18:40:23.826292 3383 ts_tablet_manager.cc:1397] T c2897d812a2e4ad79b0c91448a4aeb06 P cde3a3270250472793136621fc039e6a: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.007s
I20260620 18:40:23.835479 3382 raft_consensus.cc:348] T c2897d812a2e4ad79b0c91448a4aeb06 P ec14a1049305490fa2a9b11d750ce8c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec14a1049305490fa2a9b11d750ce8c1" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 37595 } } peers { permanent_uuid: "193a123616694962b2cd4044f703bb83" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 33547 } } peers { permanent_uuid: "cde3a3270250472793136621fc039e6a" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 32929 } }
I20260620 18:40:23.835911 3382 raft_consensus.cc:374] T c2897d812a2e4ad79b0c91448a4aeb06 P ec14a1049305490fa2a9b11d750ce8c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:40:23.836004 3382 raft_consensus.cc:729] T c2897d812a2e4ad79b0c91448a4aeb06 P ec14a1049305490fa2a9b11d750ce8c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec14a1049305490fa2a9b11d750ce8c1, State: Initialized, Role: FOLLOWER
I20260620 18:40:23.836604 3382 consensus_queue.cc:260] T c2897d812a2e4ad79b0c91448a4aeb06 P ec14a1049305490fa2a9b11d750ce8c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec14a1049305490fa2a9b11d750ce8c1" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 37595 } } peers { permanent_uuid: "193a123616694962b2cd4044f703bb83" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 33547 } } peers { permanent_uuid: "cde3a3270250472793136621fc039e6a" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 32929 } }
I20260620 18:40:23.836730 3384 raft_consensus.cc:348] T c2897d812a2e4ad79b0c91448a4aeb06 P 193a123616694962b2cd4044f703bb83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec14a1049305490fa2a9b11d750ce8c1" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 37595 } } peers { permanent_uuid: "193a123616694962b2cd4044f703bb83" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 33547 } } peers { permanent_uuid: "cde3a3270250472793136621fc039e6a" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 32929 } }
I20260620 18:40:23.837201 3384 raft_consensus.cc:374] T c2897d812a2e4ad79b0c91448a4aeb06 P 193a123616694962b2cd4044f703bb83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:40:23.836941 3383 raft_consensus.cc:348] T c2897d812a2e4ad79b0c91448a4aeb06 P cde3a3270250472793136621fc039e6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec14a1049305490fa2a9b11d750ce8c1" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 37595 } } peers { permanent_uuid: "193a123616694962b2cd4044f703bb83" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 33547 } } peers { permanent_uuid: "cde3a3270250472793136621fc039e6a" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 32929 } }
I20260620 18:40:23.837320 3384 raft_consensus.cc:729] T c2897d812a2e4ad79b0c91448a4aeb06 P 193a123616694962b2cd4044f703bb83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 193a123616694962b2cd4044f703bb83, State: Initialized, Role: FOLLOWER
I20260620 18:40:23.837373 3383 raft_consensus.cc:374] T c2897d812a2e4ad79b0c91448a4aeb06 P cde3a3270250472793136621fc039e6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:40:23.837478 3383 raft_consensus.cc:729] T c2897d812a2e4ad79b0c91448a4aeb06 P cde3a3270250472793136621fc039e6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cde3a3270250472793136621fc039e6a, State: Initialized, Role: FOLLOWER
I20260620 18:40:23.837939 3384 consensus_queue.cc:260] T c2897d812a2e4ad79b0c91448a4aeb06 P 193a123616694962b2cd4044f703bb83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec14a1049305490fa2a9b11d750ce8c1" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 37595 } } peers { permanent_uuid: "193a123616694962b2cd4044f703bb83" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 33547 } } peers { permanent_uuid: "cde3a3270250472793136621fc039e6a" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 32929 } }
I20260620 18:40:23.838195 3229 heartbeater.cc:503] Master 127.23.115.62:35261 was elected leader, sending a full tablet report...
I20260620 18:40:23.838084 3383 consensus_queue.cc:260] T c2897d812a2e4ad79b0c91448a4aeb06 P cde3a3270250472793136621fc039e6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec14a1049305490fa2a9b11d750ce8c1" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 37595 } } peers { permanent_uuid: "193a123616694962b2cd4044f703bb83" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 33547 } } peers { permanent_uuid: "cde3a3270250472793136621fc039e6a" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 32929 } }
I20260620 18:40:23.838780 3382 ts_tablet_manager.cc:1428] T c2897d812a2e4ad79b0c91448a4aeb06 P ec14a1049305490fa2a9b11d750ce8c1: Time spent starting tablet: real 0.013s user 0.010s sys 0.004s
I20260620 18:40:23.842677 3383 ts_tablet_manager.cc:1428] T c2897d812a2e4ad79b0c91448a4aeb06 P cde3a3270250472793136621fc039e6a: Time spent starting tablet: real 0.016s user 0.012s sys 0.005s
I20260620 18:40:23.843628 3363 heartbeater.cc:503] Master 127.23.115.62:35261 was elected leader, sending a full tablet report...
I20260620 18:40:23.844901 3384 ts_tablet_manager.cc:1428] T c2897d812a2e4ad79b0c91448a4aeb06 P 193a123616694962b2cd4044f703bb83: Time spent starting tablet: real 0.019s user 0.007s sys 0.012s
I20260620 18:40:23.849576 3390 raft_consensus.cc:482] T c2897d812a2e4ad79b0c91448a4aeb06 P cde3a3270250472793136621fc039e6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:40:23.849844 3390 raft_consensus.cc:504] T c2897d812a2e4ad79b0c91448a4aeb06 P cde3a3270250472793136621fc039e6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec14a1049305490fa2a9b11d750ce8c1" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 37595 } } peers { permanent_uuid: "193a123616694962b2cd4044f703bb83" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 33547 } } peers { permanent_uuid: "cde3a3270250472793136621fc039e6a" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 32929 } }
W20260620 18:40:23.850512 3230 tablet.cc:2369] T c2897d812a2e4ad79b0c91448a4aeb06 P ec14a1049305490fa2a9b11d750ce8c1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:40:23.851187 3388 raft_consensus.cc:482] T c2897d812a2e4ad79b0c91448a4aeb06 P ec14a1049305490fa2a9b11d750ce8c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:40:23.851425 3388 raft_consensus.cc:504] T c2897d812a2e4ad79b0c91448a4aeb06 P ec14a1049305490fa2a9b11d750ce8c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec14a1049305490fa2a9b11d750ce8c1" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 37595 } } peers { permanent_uuid: "193a123616694962b2cd4044f703bb83" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 33547 } } peers { permanent_uuid: "cde3a3270250472793136621fc039e6a" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 32929 } }
I20260620 18:40:23.851814 3390 leader_election.cc:288] T c2897d812a2e4ad79b0c91448a4aeb06 P cde3a3270250472793136621fc039e6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec14a1049305490fa2a9b11d750ce8c1 (127.23.115.2:37595), 193a123616694962b2cd4044f703bb83 (127.23.115.3:33547)
I20260620 18:40:23.853204 3388 leader_election.cc:288] T c2897d812a2e4ad79b0c91448a4aeb06 P ec14a1049305490fa2a9b11d750ce8c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 193a123616694962b2cd4044f703bb83 (127.23.115.3:33547), cde3a3270250472793136621fc039e6a (127.23.115.1:32929)
I20260620 18:40:23.860924 3379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:23.852232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.1:40925 (local address 127.23.115.3:33547)
0620 18:40:23.852418 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:23.852425 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:23.853609 (+ 1184us) server_negotiation.cc:407] Connection header received
0620 18:40:23.853793 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:23.853798 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:23.853887 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:23.853982 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:40:23.854837 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:23.855978 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:23.858629 (+ 2651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:23.859821 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:23.859918 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:23.859978 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:23.860540 (+ 562us) server_negotiation.cc:299] Negotiation successful
0620 18:40:23.860673 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 18:40:23.861372 3391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:23.851991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.2:37595 (local address 127.23.115.1:32809)
0620 18:40:23.852895 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:23.852912 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:40:23.853048 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:40:23.854211 (+ 1163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:40:23.854216 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:40:23.854242 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:40:23.854803 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:23.854819 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:23.856378 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:23.856383 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:23.858458 (+ 2075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:23.858473 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:23.860313 (+ 1840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:23.860319 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:23.860509 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:23.860537 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 18:40:23.860608 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 18:40:23.860743 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":775,"mutex_wait_us":44,"thread_start_us":711,"threads_started":1}
I20260620 18:40:23.862859 3378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:23.852504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.1:32809 (local address 127.23.115.2:37595)
0620 18:40:23.852701 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:23.852707 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:23.852992 (+ 285us) server_negotiation.cc:407] Connection header received
0620 18:40:23.853833 (+ 841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:23.853839 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:23.853936 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:23.854010 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:40:23.855149 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:23.856211 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:23.858648 (+ 2437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:23.860107 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:23.860262 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:23.860334 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:23.862039 (+ 1705us) server_negotiation.cc:299] Negotiation successful
0620 18:40:23.862591 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":62}
I20260620 18:40:23.862895 3392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:23.852265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.3:33547 (local address 127.23.115.1:40925)
0620 18:40:23.853508 (+ 1243us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:23.853524 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:40:23.853656 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:40:23.854151 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:40:23.854156 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:40:23.854185 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:40:23.854621 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:23.854633 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:23.856148 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:23.856153 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:23.858488 (+ 2335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:23.858498 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:23.860122 (+ 1624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:23.860126 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:23.860318 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:23.860393 (+ 75us) client_negotiation.cc:769] Sending connection context
0620 18:40:23.861813 (+ 1420us) client_negotiation.cc:240] Negotiation successful
0620 18:40:23.862499 (+ 686us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1109,"mutex_wait_us":69,"thread_start_us":974,"threads_started":1}
I20260620 18:40:23.863817 3184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2897d812a2e4ad79b0c91448a4aeb06" candidate_uuid: "cde3a3270250472793136621fc039e6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec14a1049305490fa2a9b11d750ce8c1" is_pre_election: true
I20260620 18:40:23.864279 3184 raft_consensus.cc:2454] T c2897d812a2e4ad79b0c91448a4aeb06 P ec14a1049305490fa2a9b11d750ce8c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cde3a3270250472793136621fc039e6a in term 0.
I20260620 18:40:23.864280 3318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2897d812a2e4ad79b0c91448a4aeb06" candidate_uuid: "cde3a3270250472793136621fc039e6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "193a123616694962b2cd4044f703bb83" is_pre_election: true
I20260620 18:40:23.864715 3318 raft_consensus.cc:2454] T c2897d812a2e4ad79b0c91448a4aeb06 P 193a123616694962b2cd4044f703bb83 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cde3a3270250472793136621fc039e6a in term 0.
I20260620 18:40:23.865078 2977 leader_election.cc:302] T c2897d812a2e4ad79b0c91448a4aeb06 P cde3a3270250472793136621fc039e6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cde3a3270250472793136621fc039e6a, ec14a1049305490fa2a9b11d750ce8c1; no voters:
I20260620 18:40:23.865324 3393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:23.853497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.3:33547 (local address 127.23.115.2:42343)
0620 18:40:23.854136 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:23.854152 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:40:23.854255 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:40:23.856125 (+ 1870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:40:23.856131 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:40:23.856170 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:40:23.856614 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:23.856624 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:23.858289 (+ 1665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:23.858295 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:23.862061 (+ 3766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:23.862079 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:23.864449 (+ 2370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:23.864456 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:23.864624 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:23.864657 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 18:40:23.864748 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 18:40:23.864916 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"mutex_wait_us":1,"thread_start_us":733,"threads_started":1}
I20260620 18:40:23.865509 3390 raft_consensus.cc:2781] T c2897d812a2e4ad79b0c91448a4aeb06 P cde3a3270250472793136621fc039e6a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:40:23.865656 3390 raft_consensus.cc:482] T c2897d812a2e4ad79b0c91448a4aeb06 P cde3a3270250472793136621fc039e6a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:40:23.865753 3390 raft_consensus.cc:3037] T c2897d812a2e4ad79b0c91448a4aeb06 P cde3a3270250472793136621fc039e6a [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:40:23.867586 3394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:23.854025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.2:42343 (local address 127.23.115.3:33547)
0620 18:40:23.855583 (+ 1558us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:23.855589 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:23.855612 (+ 23us) server_negotiation.cc:407] Connection header received
0620 18:40:23.855669 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:23.855674 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:23.855744 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:23.855844 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:40:23.856803 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:23.858123 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:23.862258 (+ 4135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:23.864058 (+ 1800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:23.864217 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:23.864289 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:23.867115 (+ 2826us) server_negotiation.cc:299] Negotiation successful
0620 18:40:23.867250 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1406,"thread_start_us":1217,"threads_started":1}
I20260620 18:40:23.869031 3390 raft_consensus.cc:504] T c2897d812a2e4ad79b0c91448a4aeb06 P cde3a3270250472793136621fc039e6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec14a1049305490fa2a9b11d750ce8c1" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 37595 } } peers { permanent_uuid: "193a123616694962b2cd4044f703bb83" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 33547 } } peers { permanent_uuid: "cde3a3270250472793136621fc039e6a" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 32929 } }
I20260620 18:40:23.869414 3318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2897d812a2e4ad79b0c91448a4aeb06" candidate_uuid: "ec14a1049305490fa2a9b11d750ce8c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "193a123616694962b2cd4044f703bb83" is_pre_election: true
I20260620 18:40:23.869629 3318 raft_consensus.cc:2454] T c2897d812a2e4ad79b0c91448a4aeb06 P 193a123616694962b2cd4044f703bb83 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec14a1049305490fa2a9b11d750ce8c1 in term 0.
I20260620 18:40:23.870028 3184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2897d812a2e4ad79b0c91448a4aeb06" candidate_uuid: "cde3a3270250472793136621fc039e6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec14a1049305490fa2a9b11d750ce8c1"
I20260620 18:40:23.870155 3118 leader_election.cc:302] T c2897d812a2e4ad79b0c91448a4aeb06 P ec14a1049305490fa2a9b11d750ce8c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 193a123616694962b2cd4044f703bb83, ec14a1049305490fa2a9b11d750ce8c1; no voters:
I20260620 18:40:23.870244 3184 raft_consensus.cc:3037] T c2897d812a2e4ad79b0c91448a4aeb06 P ec14a1049305490fa2a9b11d750ce8c1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:40:23.870288 3318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2897d812a2e4ad79b0c91448a4aeb06" candidate_uuid: "cde3a3270250472793136621fc039e6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "193a123616694962b2cd4044f703bb83"
I20260620 18:40:23.870452 3318 raft_consensus.cc:3037] T c2897d812a2e4ad79b0c91448a4aeb06 P 193a123616694962b2cd4044f703bb83 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:40:23.873396 3381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:23.855027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.2:56459 (local address 127.23.115.1:32929)
0620 18:40:23.855192 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:23.855198 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:23.868017 (+ 12819us) server_negotiation.cc:407] Connection header received
0620 18:40:23.868193 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:23.868198 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:23.868268 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:23.868347 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:40:23.868826 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:23.869852 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:23.871783 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:23.872569 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:23.872711 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:23.872782 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:23.873054 (+ 272us) server_negotiation.cc:299] Negotiation successful
0620 18:40:23.873174 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 18:40:23.873497 3318 raft_consensus.cc:2454] T c2897d812a2e4ad79b0c91448a4aeb06 P 193a123616694962b2cd4044f703bb83 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cde3a3270250472793136621fc039e6a in term 1.
I20260620 18:40:23.873507 3184 raft_consensus.cc:2454] T c2897d812a2e4ad79b0c91448a4aeb06 P ec14a1049305490fa2a9b11d750ce8c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cde3a3270250472793136621fc039e6a in term 1.
I20260620 18:40:23.873510 3393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:23.854665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.1:32929 (local address 127.23.115.2:56459)
0620 18:40:23.867931 (+ 13266us) negotiation.cc:107] Waiting for socket to connect
0620 18:40:23.867950 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:40:23.868088 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:40:23.868498 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:40:23.868502 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:40:23.868529 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:40:23.868669 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:23.868679 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:23.869996 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:23.870001 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:23.871635 (+ 1634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:40:23.871669 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:40:23.872711 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:40:23.872717 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:40:23.872900 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:23.872924 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 18:40:23.873000 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 18:40:23.873144 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13170,"thread_start_us":14098,"threads_started":1}
I20260620 18:40:23.873677 3388 raft_consensus.cc:2741] T c2897d812a2e4ad79b0c91448a4aeb06 P ec14a1049305490fa2a9b11d750ce8c1 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260620 18:40:23.874071 2975 leader_election.cc:302] T c2897d812a2e4ad79b0c91448a4aeb06 P cde3a3270250472793136621fc039e6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 193a123616694962b2cd4044f703bb83, cde3a3270250472793136621fc039e6a; no voters:
I20260620 18:40:23.874778 3390 leader_election.cc:288] T c2897d812a2e4ad79b0c91448a4aeb06 P cde3a3270250472793136621fc039e6a [CANDIDATE]: Term 1 election: Requested vote from peers ec14a1049305490fa2a9b11d750ce8c1 (127.23.115.2:37595), 193a123616694962b2cd4044f703bb83 (127.23.115.3:33547)
I20260620 18:40:23.874617 3041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2897d812a2e4ad79b0c91448a4aeb06" candidate_uuid: "ec14a1049305490fa2a9b11d750ce8c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cde3a3270250472793136621fc039e6a" is_pre_election: true
I20260620 18:40:23.875003 3390 raft_consensus.cc:2781] T c2897d812a2e4ad79b0c91448a4aeb06 P cde3a3270250472793136621fc039e6a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:40:23.875118 3390 raft_consensus.cc:686] T c2897d812a2e4ad79b0c91448a4aeb06 P cde3a3270250472793136621fc039e6a [term 1 LEADER]: Becoming Leader. State: Replica: cde3a3270250472793136621fc039e6a, State: Running, Role: LEADER
I20260620 18:40:23.875552 3390 consensus_queue.cc:237] T c2897d812a2e4ad79b0c91448a4aeb06 P cde3a3270250472793136621fc039e6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec14a1049305490fa2a9b11d750ce8c1" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 37595 } } peers { permanent_uuid: "193a123616694962b2cd4044f703bb83" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 33547 } } peers { permanent_uuid: "cde3a3270250472793136621fc039e6a" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 32929 } }
I20260620 18:40:23.882722 2871 catalog_manager.cc:5495] T c2897d812a2e4ad79b0c91448a4aeb06 P cde3a3270250472793136621fc039e6a reported cstate change: term changed from 0 to 1, leader changed from <none> to cde3a3270250472793136621fc039e6a (127.23.115.1). New cstate: current_term: 1 leader_uuid: "cde3a3270250472793136621fc039e6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec14a1049305490fa2a9b11d750ce8c1" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 37595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "193a123616694962b2cd4044f703bb83" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 33547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cde3a3270250472793136621fc039e6a" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 32929 } health_report { overall_health: HEALTHY } } }
W20260620 18:40:23.894388 3088 tablet.cc:2369] T c2897d812a2e4ad79b0c91448a4aeb06 P cde3a3270250472793136621fc039e6a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:40:23.904878 3381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:23.899385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59234 (local address 127.23.115.1:32929)
0620 18:40:23.899590 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:23.899596 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:23.899667 (+ 71us) server_negotiation.cc:407] Connection header received
0620 18:40:23.899877 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:23.899880 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:23.900114 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:23.900272 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:40:23.900641 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:23.901354 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:23.902739 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:23.903154 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:23.903269 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:23.903784 (+ 515us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:40:23.904296 (+ 512us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:40:23.904422 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:23.904520 (+ 98us) server_negotiation.cc:299] Negotiation successful
0620 18:40:23.904598 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260620 18:40:23.912833 3184 raft_consensus.cc:1261] T c2897d812a2e4ad79b0c91448a4aeb06 P ec14a1049305490fa2a9b11d750ce8c1 [term 1 FOLLOWER]: Refusing update from remote peer cde3a3270250472793136621fc039e6a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:40:23.912943 3318 raft_consensus.cc:1261] T c2897d812a2e4ad79b0c91448a4aeb06 P 193a123616694962b2cd4044f703bb83 [term 1 FOLLOWER]: Refusing update from remote peer cde3a3270250472793136621fc039e6a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:40:23.913692 3396 consensus_queue.cc:1035] T c2897d812a2e4ad79b0c91448a4aeb06 P cde3a3270250472793136621fc039e6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec14a1049305490fa2a9b11d750ce8c1" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 37595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:40:23.914093 3390 consensus_queue.cc:1035] T c2897d812a2e4ad79b0c91448a4aeb06 P cde3a3270250472793136621fc039e6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "193a123616694962b2cd4044f703bb83" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 33547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:40:23.942737 2871 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35344:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:40:23.943050 2871 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35344:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:40:23.946709 2871 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c5c4212f13c34ea6b649d74194dde649: Sending DeleteTablet for 3 replicas of tablet c2897d812a2e4ad79b0c91448a4aeb06
I20260620 18:40:23.947892 3164 tablet_service.cc:1514] Processing DeleteTablet for tablet c2897d812a2e4ad79b0c91448a4aeb06 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:40:23 UTC) from {username='kudu', principal='kudu/127.23.115.62@KRBTEST.COM'} at 127.0.0.1:46342
I20260620 18:40:23.948004 3298 tablet_service.cc:1514] Processing DeleteTablet for tablet c2897d812a2e4ad79b0c91448a4aeb06 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:40:23 UTC) from {username='kudu', principal='kudu/127.23.115.62@KRBTEST.COM'} at 127.0.0.1:41692
I20260620 18:40:23.948204 3021 tablet_service.cc:1514] Processing DeleteTablet for tablet c2897d812a2e4ad79b0c91448a4aeb06 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:40:23 UTC) from {username='kudu', principal='kudu/127.23.115.62@KRBTEST.COM'} at 127.0.0.1:59232
I20260620 18:40:23.950610 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 2953
I20260620 18:40:23.960379 3411 tablet_replica.cc:330] T c2897d812a2e4ad79b0c91448a4aeb06 P cde3a3270250472793136621fc039e6a: stopping tablet replica
I20260620 18:40:23.961189 3410 tablet_replica.cc:330] T c2897d812a2e4ad79b0c91448a4aeb06 P ec14a1049305490fa2a9b11d750ce8c1: stopping tablet replica
I20260620 18:40:23.961526 3411 raft_consensus.cc:2229] T c2897d812a2e4ad79b0c91448a4aeb06 P cde3a3270250472793136621fc039e6a [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:40:23.961586 3410 raft_consensus.cc:2229] T c2897d812a2e4ad79b0c91448a4aeb06 P ec14a1049305490fa2a9b11d750ce8c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:40:23.962095 3410 raft_consensus.cc:2258] T c2897d812a2e4ad79b0c91448a4aeb06 P ec14a1049305490fa2a9b11d750ce8c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:40:23.962356 3411 raft_consensus.cc:2258] T c2897d812a2e4ad79b0c91448a4aeb06 P cde3a3270250472793136621fc039e6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:40:23.962846 3409 tablet_replica.cc:330] T c2897d812a2e4ad79b0c91448a4aeb06 P 193a123616694962b2cd4044f703bb83: stopping tablet replica
I20260620 18:40:23.963259 3409 raft_consensus.cc:2229] T c2897d812a2e4ad79b0c91448a4aeb06 P 193a123616694962b2cd4044f703bb83 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:40:23.963692 3409 raft_consensus.cc:2258] T c2897d812a2e4ad79b0c91448a4aeb06 P 193a123616694962b2cd4044f703bb83 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:40:23.964679 3410 ts_tablet_manager.cc:1905] T c2897d812a2e4ad79b0c91448a4aeb06 P ec14a1049305490fa2a9b11d750ce8c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:40:23.964690 3411 ts_tablet_manager.cc:1905] T c2897d812a2e4ad79b0c91448a4aeb06 P cde3a3270250472793136621fc039e6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:40:23.972288 3411 ts_tablet_manager.cc:1918] T c2897d812a2e4ad79b0c91448a4aeb06 P cde3a3270250472793136621fc039e6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:40:23.972575 3411 log.cc:1196] T c2897d812a2e4ad79b0c91448a4aeb06 P cde3a3270250472793136621fc039e6a: Deleting WAL directory at /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/minicluster-data/ts-0/wal/wals/c2897d812a2e4ad79b0c91448a4aeb06
I20260620 18:40:23.972698 3409 ts_tablet_manager.cc:1905] T c2897d812a2e4ad79b0c91448a4aeb06 P 193a123616694962b2cd4044f703bb83: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 20 18:40:23 dist-test-slave-d645 krb5kdc[2397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980819, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.115.1@KRBTEST.COM
I20260620 18:40:23.973337 3411 ts_tablet_manager.cc:1939] T c2897d812a2e4ad79b0c91448a4aeb06 P cde3a3270250472793136621fc039e6a: Deleting consensus metadata
I20260620 18:40:23.975281 2858 catalog_manager.cc:4841] TS cde3a3270250472793136621fc039e6a (127.23.115.1:32929): tablet c2897d812a2e4ad79b0c91448a4aeb06 (table test-table [id=0df382dc85c84e1093b9502988a8cb00]) successfully deleted
I20260620 18:40:23.978320 3409 ts_tablet_manager.cc:1918] T c2897d812a2e4ad79b0c91448a4aeb06 P 193a123616694962b2cd4044f703bb83: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:40:23.978463 3409 log.cc:1196] T c2897d812a2e4ad79b0c91448a4aeb06 P 193a123616694962b2cd4044f703bb83: Deleting WAL directory at /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/minicluster-data/ts-2/wal/wals/c2897d812a2e4ad79b0c91448a4aeb06
I20260620 18:40:23.978966 3410 ts_tablet_manager.cc:1918] T c2897d812a2e4ad79b0c91448a4aeb06 P ec14a1049305490fa2a9b11d750ce8c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:40:23.979012 3409 ts_tablet_manager.cc:1939] T c2897d812a2e4ad79b0c91448a4aeb06 P 193a123616694962b2cd4044f703bb83: Deleting consensus metadata
I20260620 18:40:23.979188 3410 log.cc:1196] T c2897d812a2e4ad79b0c91448a4aeb06 P ec14a1049305490fa2a9b11d750ce8c1: Deleting WAL directory at /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980676253068-24012-0/minicluster-data/ts-1/wal/wals/c2897d812a2e4ad79b0c91448a4aeb06
I20260620 18:40:23.979729 3410 ts_tablet_manager.cc:1939] T c2897d812a2e4ad79b0c91448a4aeb06 P ec14a1049305490fa2a9b11d750ce8c1: Deleting consensus metadata
I20260620 18:40:23.981173 2858 catalog_manager.cc:4841] TS 193a123616694962b2cd4044f703bb83 (127.23.115.3:33547): tablet c2897d812a2e4ad79b0c91448a4aeb06 (table test-table [id=0df382dc85c84e1093b9502988a8cb00]) successfully deleted
I20260620 18:40:23.981225 2860 catalog_manager.cc:4841] TS ec14a1049305490fa2a9b11d750ce8c1 (127.23.115.2:37595): tablet c2897d812a2e4ad79b0c91448a4aeb06 (table test-table [id=0df382dc85c84e1093b9502988a8cb00]) successfully deleted
I20260620 18:40:23.982208 3381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:23.953550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59242 (local address 127.23.115.1:32929)
0620 18:40:23.953725 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:23.953732 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:23.960547 (+ 6815us) server_negotiation.cc:407] Connection header received
0620 18:40:23.960753 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:23.960759 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:23.960883 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:23.960982 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:40:23.964421 (+ 3439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:23.965504 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:23.967990 (+ 2486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:23.968693 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:23.968795 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:23.974409 (+ 5614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:40:23.974424 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:40:23.974446 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:40:23.974504 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:40:23.978534 (+ 4030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:40:23.979517 (+ 983us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:40:23.979526 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:40:23.979530 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:40:23.979709 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:40:23.980191 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:40:23.980197 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:40:23.980200 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:40:23.980814 (+ 614us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:40:23.981148 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:23.981795 (+ 647us) server_negotiation.cc:299] Negotiation successful
0620 18:40:23.981973 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 18:40:23.983003 3082 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:40:24.082420 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 2953
I20260620 18:40:24.101178 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 3094
Jun 20 18:40:24 dist-test-slave-d645 krb5kdc[2397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980819, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.115.2@KRBTEST.COM
I20260620 18:40:24.116519 3378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:24.101873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46354 (local address 127.23.115.2:37595)
0620 18:40:24.102057 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:24.102062 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:24.102173 (+ 111us) server_negotiation.cc:407] Connection header received
0620 18:40:24.102330 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:24.102335 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:24.102409 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:24.102486 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:40:24.103765 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:24.104509 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:24.105742 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:24.106123 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:24.106214 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:24.110450 (+ 4236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:40:24.110464 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:40:24.110480 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:40:24.110519 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:40:24.113566 (+ 3047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:40:24.114319 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:40:24.114327 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:40:24.114329 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:40:24.114482 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:40:24.114878 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:40:24.114882 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:40:24.114884 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:40:24.115323 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:40:24.115553 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:24.115816 (+ 263us) server_negotiation.cc:299] Negotiation successful
0620 18:40:24.116017 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 18:40:24.117192 3224 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:40:24.208549 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 3094
I20260620 18:40:24.222879 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 3235
Jun 20 18:40:24 dist-test-slave-d645 krb5kdc[2397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980819, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.115.3@KRBTEST.COM
I20260620 18:40:24.238893 3394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:40:24.223898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41700 (local address 127.23.115.3:33547)
0620 18:40:24.224263 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0620 18:40:24.224268 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:40:24.224288 (+ 20us) server_negotiation.cc:407] Connection header received
0620 18:40:24.224346 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:40:24.224350 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:40:24.224418 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:40:24.224525 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:40:24.225937 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:24.226683 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:24.228157 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:40:24.228572 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:40:24.228692 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:40:24.233036 (+ 4344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:40:24.233053 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:40:24.233070 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:40:24.233108 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:40:24.235785 (+ 2677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:40:24.236512 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:40:24.236519 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:40:24.236521 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:40:24.236650 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:40:24.237398 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:40:24.237405 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:40:24.237408 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:40:24.237801 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:40:24.238029 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0620 18:40:24.238395 (+ 366us) server_negotiation.cc:299] Negotiation successful
0620 18:40:24.238570 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260620 18:40:24.239560 3358 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:40:24.331048 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 3235
I20260620 18:40:24.344622 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 2841
I20260620 18:40:24.345697 2903 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:40:24.457255 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 2841
I20260620 18:40:24.479789 24012 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 18:40:24.815883 24012 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 18:40:24.816118 24012 mini_ranger.cc:66] Stopping Ranger...
I20260620 18:40:25.153060 24012 mini_ranger.cc:68] Stopped Ranger
2026-06-20 18:40:25.153 UTC [2427] LOG: received smart shutdown request
2026-06-20 18:40:25.156 UTC [2427] LOG: background worker "logical replication launcher" (PID 2436) exited with exit code 1
2026-06-20 18:40:25.156 UTC [2431] LOG: shutting down
2026-06-20 18:40:25.350 UTC [2427] LOG: database system is shut down
2026-06-20T18:40:25Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (52430 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:40:25 dist-test-slave-d645 krb5kdc[3425](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:40:25 dist-test-slave-d645 krb5kdc[3425](info): set up 2 sockets
Jun 20 18:40:25 dist-test-slave-d645 krb5kdc[3425](info): commencing operation
krb5kdc: starting...
W20260620 18:40:26.511512 24012 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/krb5kdc/test-user.keytab.
Jun 20 18:40:26 dist-test-slave-d645 krb5kdc[3425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980826, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:40:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:40:26Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.23.115.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.23.115.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.23.115.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/krb5kdc/rangeradmin_127.23.115.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.23.115.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/krb5kdc/rangeradmin_127.23.115.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.23.115.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.23.115.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.23.115.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/krb5kdc/rangerlookup_127.23.115.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.23.115.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/krb5kdc/rangerlookup_127.23.115.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.23.115.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.23.115.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.115.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/krb5kdc/HTTP_127.23.115.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.23.115.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/krb5kdc/HTTP_127.23.115.20@KRBTEST.COM.keytab.
I20260620 18:40:26.838584 24012 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task6x5P9o/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/postgres -l logfile start
2026-06-20 18:40:29.666 UTC [3455] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 18:40:29.666 UTC [3455] LOG: listening on IPv4 address "127.23.115.20", port 37359
2026-06-20 18:40:29.670 UTC [3455] LOG: listening on Unix socket "/tmp/.s.PGSQL.37359"
2026-06-20 18:40:29.692 UTC [3458] LOG: database system was shut down at 2026-06-20 18:40:27 UTC
2026-06-20 18:40:29.697 UTC [3455] LOG: database system is ready to accept connections
I20260620 18:40:30.595027 24012 mini_postgres.cc:96] Postgres bound to 37359
2026-06-20 18:40:30.611 UTC [3466] FATAL: database "slave" does not exist
127.23.115.20:37359 - accepting connections
I20260620 18:40:30.612628 24012 mini_ranger.cc:161] Starting Ranger...
I20260620 18:40:30.642179 24012 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 18:40:31.271304 24012 mini_ranger.cc:87] Created ranger Postgres database
I20260620 18:40:31.271420 24012 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ranger-admin
2026-06-20 18:40:31,560 [I] DB FLAVOR :POSTGRES
2026-06-20 18:40:31,560 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 18:40:31,560 [I] Checking connection..
2026-06-20 18:40:31,561 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6x5P9o/build/asan/bin/postgresql.jar:/tmp/dist-test-task6x5P9o/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.115.20:37359/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 18:40:31,832 [I] Checking connection passed.
2026-06-20 18:40:31,832 [I] --------- Verifying version history table ---------
2026-06-20 18:40:31,832 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6x5P9o/build/asan/bin/postgresql.jar:/tmp/dist-test-task6x5P9o/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.115.20:37359/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 18:40:32,097 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 18:40:32,098 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6x5P9o/build/asan/bin/postgresql.jar:/tmp/dist-test-task6x5P9o/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.115.20:37359/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 18:40:32,341 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 18:40:32,342 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 18:40:32,342 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6x5P9o/build/asan/bin/postgresql.jar:/tmp/dist-test-task6x5P9o/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.115.20:37359/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task6x5P9o/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 18:40:32.603 UTC [3530] WARNING: there is no transaction in progress
2026-06-20 18:40:32,609 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 18:40:32,609 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6x5P9o/build/asan/bin/postgresql.jar:/tmp/dist-test-task6x5P9o/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.115.20:37359/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 18:40:32,865 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 18:40:32,865 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 18:40:32,865 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6x5P9o/build/asan/bin/postgresql.jar:/tmp/dist-test-task6x5P9o/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.115.20:37359/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 18:40:33,132 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6x5P9o/build/asan/bin/postgresql.jar:/tmp/dist-test-task6x5P9o/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.115.20:37359/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 18:40:33,379 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6x5P9o/build/asan/bin/postgresql.jar:/tmp/dist-test-task6x5P9o/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.115.20:37359/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-d645.c.gcp-upstream.internal','N') ;"
2026-06-20 18:40:33,626 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6x5P9o/build/asan/bin/postgresql.jar:/tmp/dist-test-task6x5P9o/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.115.20:37359/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 18:40:33,895 [I] Table x_portal_user does not exist in database ranger
2026-06-20 18:40:33,895 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6x5P9o/build/asan/bin/postgresql.jar:/tmp/dist-test-task6x5P9o/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.115.20:37359/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 18:40:34,173 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 18:40:34,174 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6x5P9o/build/asan/bin/postgresql.jar:/tmp/dist-test-task6x5P9o/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.115.20:37359/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 18:40:34,444 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6x5P9o/build/asan/bin/postgresql.jar:/tmp/dist-test-task6x5P9o/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.115.20:37359/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 18:40:34,679 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 18:40:34,679 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6x5P9o/build/asan/bin/postgresql.jar:/tmp/dist-test-task6x5P9o/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.115.20:37359/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task6x5P9o/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 18:40:35.719 UTC [3638] WARNING: there is no transaction in progress
2026-06-20 18:40:35.737 UTC [3638] WARNING: there is no transaction in progress
2026-06-20 18:40:35.756 UTC [3638] WARNING: there is no transaction in progress
2026-06-20 18:40:35.773 UTC [3638] WARNING: there is no transaction in progress
2026-06-20 18:40:35.790 UTC [3638] WARNING: there is no transaction in progress
2026-06-20 18:40:35.810 UTC [3638] WARNING: there is no transaction in progress
2026-06-20 18:40:35.882 UTC [3638] WARNING: there is no transaction in progress
2026-06-20 18:40:35.891 UTC [3638] WARNING: there is no transaction in progress
2026-06-20 18:40:35.911 UTC [3638] WARNING: there is no transaction in progress
2026-06-20 18:40:35.924 UTC [3638] WARNING: there is no transaction in progress
2026-06-20 18:40:35.935 UTC [3638] WARNING: there is no transaction in progress
2026-06-20 18:40:35.951 UTC [3638] WARNING: there is no transaction in progress
2026-06-20 18:40:35.963 UTC [3638] WARNING: there is no transaction in progress
2026-06-20 18:40:35.971 UTC [3638] WARNING: there is no transaction in progress
2026-06-20 18:40:35.980 UTC [3638] WARNING: there is no transaction in progress
2026-06-20 18:40:35.989 UTC [3638] WARNING: there is no transaction in progress
2026-06-20 18:40:36,778 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 18:40:36,778 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6x5P9o/build/asan/bin/postgresql.jar:/tmp/dist-test-task6x5P9o/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.115.20:37359/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 18:40:37,050 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 18:40:37,050 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6x5P9o/build/asan/bin/postgresql.jar:/tmp/dist-test-task6x5P9o/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.115.20:37359/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 18:40:37,334 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 18:40:37,334 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6x5P9o/build/asan/bin/postgresql.jar:/tmp/dist-test-task6x5P9o/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.115.20:37359/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 18:40:37,585 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 18:40:37,585 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6x5P9o/build/asan/bin/postgresql.jar:/tmp/dist-test-task6x5P9o/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.115.20:37359/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 18:40:37,815 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6x5P9o/build/asan/bin/postgresql.jar:/tmp/dist-test-task6x5P9o/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.115.20:37359/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 18:40:38,080 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6x5P9o/build/asan/bin/postgresql.jar:/tmp/dist-test-task6x5P9o/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.115.20:37359/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-d645.c.gcp-upstream.internal';"
2026-06-20 18:40:38,328 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 18:40:38,328 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6x5P9o/build/asan/bin/postgresql.jar:/tmp/dist-test-task6x5P9o/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.115.20:37359/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 18:40:38,572 [I] DB_PATCHES have already been applied
I20260620 18:40:38.578763 24012 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ranger-admin:/tmp/dist-test-task6x5P9o/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task6x5P9o/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task6x5P9o/build/asan/bin/postgresql.jar:/tmp/dist-test-task6x5P9o/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 18:40:38.578871 24012 mini_ranger.cc:193] Using host: 127.23.115.20
I20260620 18:40:38.591935 24012 mini_ranger.cc:227] Ranger admin URL: http://127.23.115.20:43617
Jun 20, 2026 6:40:38 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 6:40:38 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 6:40:39 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 6:40:39 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 6:40:39 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 6:40:39 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 6:40:39 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 18:40:39.612323 24012 mini_ranger.cc:160] Time spent starting Ranger: real 9.000s user 0.004s sys 0.036s
Jun 20, 2026 6:41:01 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 6:41:01 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Jun 20, 2026 6:41:01 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 6:41:01 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 6:41:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 6:41:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 6:41:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 6:41:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 6:41:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 6:41:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 6:41:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 6:41:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 6:41:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 6:41:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 6:41:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 6:41:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 6:41:01 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 6:41:02 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 18:41:02.937494 24012 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.23.115.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.23.115.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.23.115.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/krb5kdc/rangerkms_127.23.115.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.23.115.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/krb5kdc/rangerkms_127.23.115.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.23.115.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.23.115.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.115.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/krb5kdc/HTTP_127.23.115.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.23.115.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/krb5kdc/HTTP_127.23.115.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 18:41:03 dist-test-slave-d645 krb5kdc[3425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980863, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 18:41:03.109778 24012 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 18:41:03.130669 24012 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 18:41:04.130134 24012 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 18:41:04.130234 24012 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ranger-kms
I20260620 18:41:04.130262 24012 mini_ranger_kms.cc:227] Using postgres at 127.23.115.20:37359
2026-06-20 18:41:04,461 [I] DB FLAVOR :POSTGRES
2026-06-20 18:41:04,461 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 18:41:04,461 [I] Checking connection
2026-06-20 18:41:04,461 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6x5P9o/build/asan/bin/postgresql.jar:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.115.20:37359/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 18:41:04,699 [I] connection success
2026-06-20 18:41:04,699 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 18:41:04,699 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 18:41:04,699 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6x5P9o/build/asan/bin/postgresql.jar:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.115.20:37359/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 18:41:04,940 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 18:41:04,940 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 18:41:04,940 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 18:41:04,940 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6x5P9o/build/asan/bin/postgresql.jar:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.115.20:37359/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 18:41:05,220 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 18:41:05.757226 24012 mini_ranger_kms.cc:310] Created kms service
I20260620 18:41:05.952716 24012 mini_ranger_kms.cc:326] Created kudu user
I20260620 18:41:06.002784 24012 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 18:41:06 dist-test-slave-d645 krb5kdc[3425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980863, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.23.115.20@KRBTEST.COM
I20260620 18:41:06.440336 24012 mini_ranger_kms.cc:384] Added ranger policy
I20260620 18:41:06.440677 24012 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ranger-kms:/tmp/dist-test-task6x5P9o/build/asan/bin/postgresql.jar:/tmp/dist-test-task6x5P9o/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task6x5P9o/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 18:41:06.440734 24012 mini_ranger_kms.cc:240] Using host: 127.23.115.20
I20260620 18:41:06.460263 24012 mini_ranger_kms.cc:276] Ranger KMS PID: 3819
I20260620 18:41:06.460378 24012 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.23.115.20:33827
Jun 20, 2026 6:41:07 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 6:41:07 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 6:41:07 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 6:41:07 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 6:41:07 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 6:41:07 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 18:41:08.595198 24012 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.485s user 0.093s sys 0.254s
I20260620 18:41:08.595474 24012 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 18:41:08.595710 24012 mini_ranger_kms.cc:401] 127.23.115.20:33827/kms/v1/keys
Jun 20 18:41:08 dist-test-slave-d645 krb5kdc[3425](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781980868, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.23.115.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 18:41:09 dist-test-slave-d645 krb5kdc[3425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980868, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.23.115.20@KRBTEST.COM for HTTP/127.23.115.20@KRBTEST.COM
Jun 20 18:41:09 dist-test-slave-d645 krb5kdc[3425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980868, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.23.115.20@KRBTEST.COM for HTTP/127.23.115.20@KRBTEST.COM
Jun 20 18:41:11 dist-test-slave-d645 krb5kdc[3425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980871, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.23.115.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.115.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.115.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab.
Entry for principal kudu/127.23.115.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab.
I20260620 18:41:11.999787 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.115.62:41203
--webserver_interface=127.23.115.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab
--principal=kudu/127.23.115.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:40323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.115.62:41203
--ranger_config_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.115.20:33827/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:41:12.340059 3869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:41:12.340354 3869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:41:12.340467 3869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:41:12.348176 3869 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:41:12.348310 3869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:41:12.348363 3869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:41:12.348440 3869 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:41:12.348500 3869 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:41:12.358851 3869 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:40323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/master-0/wal
--ranger_kms_url=127.23.115.20:33827/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.23.115.62:41203
--ranger_config_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab
--principal=kudu/127.23.115.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.115.62:41203
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.115.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:41:12.360220 3869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:41:12.361482 3869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:41:12.375001 3874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:12.374905 3875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:12.375712 3877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:41:12.376629 3869 server_base.cc:1034] running on GCE node
I20260620 18:41:12.377451 3869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:41:12.379092 3869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:41:12.380340 3869 hybrid_clock.cc:648] HybridClock initialized: now 1781980872380278 us; error 52 us; skew 500 ppm
I20260620 18:41:12.380738 3869 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:41:12 dist-test-slave-d645 krb5kdc[3425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980872, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:41:12.387848 3869 init.cc:377] Logged in from keytab as kudu/127.23.115.62@KRBTEST.COM (short username kudu)
I20260620 18:41:12.389442 3869 webserver.cc:460] Webserver started at http://127.23.115.62:44517/ using document root <none> and password file <none>
I20260620 18:41:12.390106 3869 fs_manager.cc:359] Metadata directory not provided
I20260620 18:41:12.390218 3869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:41:12.390514 3869 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 18:41:12 dist-test-slave-d645 krb5kdc[3425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980872, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.62@KRBTEST.COM for HTTP/127.23.115.20@KRBTEST.COM
I20260620 18:41:12.452531 3869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/master-0/data/instance:
uuid: "1be16d054bca4d19824b61372c2acb9e"
format_stamp: "Formatted at 2026-06-20 18:41:12 on dist-test-slave-d645"
server_key: "96a535bc1032c2a7ba375b3554f46afe"
server_key_iv: "5e328a4c2cad679b700455936db9651e"
server_key_version: "kuduclusterkey@0"
I20260620 18:41:12.453370 3869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/master-0/wal/instance:
uuid: "1be16d054bca4d19824b61372c2acb9e"
format_stamp: "Formatted at 2026-06-20 18:41:12 on dist-test-slave-d645"
server_key: "96a535bc1032c2a7ba375b3554f46afe"
server_key_iv: "5e328a4c2cad679b700455936db9651e"
server_key_version: "kuduclusterkey@0"
I20260620 18:41:12.458719 3869 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 18:41:12.480588 3884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:12.482429 3869 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 18:41:12.482661 3869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/master-0/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/master-0/wal
uuid: "1be16d054bca4d19824b61372c2acb9e"
format_stamp: "Formatted at 2026-06-20 18:41:12 on dist-test-slave-d645"
server_key: "96a535bc1032c2a7ba375b3554f46afe"
server_key_iv: "5e328a4c2cad679b700455936db9651e"
server_key_version: "kuduclusterkey@0"
I20260620 18:41:12.483039 3869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:41:12.525213 3869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:41:12.531424 3869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:41:12.531790 3869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:41:12.551862 3869 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.62:41203
I20260620 18:41:12.551834 3935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.62:41203 every 8 connection(s)
I20260620 18:41:12.553646 3869 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
I20260620 18:41:12.556209 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 3869
I20260620 18:41:12.556432 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/master-0/wal/instance
I20260620 18:41:12.557252 24012 external_mini_cluster.cc:1428] Setting key bc8f1f963a18e88d901d711f7ede40d4
I20260620 18:41:12.558350 3936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:41:12.575642 3936 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1be16d054bca4d19824b61372c2acb9e: Bootstrap starting.
I20260620 18:41:12.582098 3936 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1be16d054bca4d19824b61372c2acb9e: Neither blocks nor log segments found. Creating new log.
I20260620 18:41:12.583472 3936 log.cc:824] T 00000000000000000000000000000000 P 1be16d054bca4d19824b61372c2acb9e: Log is configured to *not* fsync() on all Append() calls
I20260620 18:41:12.586791 3936 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1be16d054bca4d19824b61372c2acb9e: No bootstrap required, opened a new log
Jun 20 18:41:12 dist-test-slave-d645 krb5kdc[3425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980871, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.115.62@KRBTEST.COM
I20260620 18:41:12.596846 3936 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1be16d054bca4d19824b61372c2acb9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1be16d054bca4d19824b61372c2acb9e" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 41203 } }
I20260620 18:41:12.597286 3936 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1be16d054bca4d19824b61372c2acb9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:41:12.597390 3936 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1be16d054bca4d19824b61372c2acb9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1be16d054bca4d19824b61372c2acb9e, State: Initialized, Role: FOLLOWER
I20260620 18:41:12.598096 3936 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1be16d054bca4d19824b61372c2acb9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1be16d054bca4d19824b61372c2acb9e" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 41203 } }
I20260620 18:41:12.598412 3936 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1be16d054bca4d19824b61372c2acb9e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:41:12.598591 3936 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1be16d054bca4d19824b61372c2acb9e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:41:12.598814 3936 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1be16d054bca4d19824b61372c2acb9e [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:41:12.600476 3939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:12.559555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46874 (local address 127.23.115.62:41203)
0620 18:41:12.574283 (+ 14728us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:12.574300 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:12.574368 (+ 68us) server_negotiation.cc:407] Connection header received
0620 18:41:12.577220 (+ 2852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:12.577265 (+ 45us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:12.577992 (+ 727us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:12.578743 (+ 751us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:12.580484 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:12.582792 (+ 2308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:12.584762 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:12.585775 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:12.585973 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:12.591187 (+ 5214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:12.591203 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:12.591230 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:12.591293 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:12.595307 (+ 4014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:12.596217 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:12.596227 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:12.596242 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:12.596489 (+ 247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:12.596953 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:12.596958 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:12.596962 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:12.597783 (+ 821us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:12.598072 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:12.598592 (+ 520us) server_negotiation.cc:299] Negotiation successful
0620 18:41:12.599048 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14441,"thread_start_us":15375,"threads_started":1}
I20260620 18:41:12.601411 3936 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1be16d054bca4d19824b61372c2acb9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1be16d054bca4d19824b61372c2acb9e" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 41203 } }
I20260620 18:41:12.601950 3936 leader_election.cc:302] T 00000000000000000000000000000000 P 1be16d054bca4d19824b61372c2acb9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1be16d054bca4d19824b61372c2acb9e; no voters:
I20260620 18:41:12.602596 3936 leader_election.cc:288] T 00000000000000000000000000000000 P 1be16d054bca4d19824b61372c2acb9e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:41:12.602777 3941 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1be16d054bca4d19824b61372c2acb9e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:41:12.605065 3941 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1be16d054bca4d19824b61372c2acb9e [term 1 LEADER]: Becoming Leader. State: Replica: 1be16d054bca4d19824b61372c2acb9e, State: Running, Role: LEADER
I20260620 18:41:12.605710 3941 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1be16d054bca4d19824b61372c2acb9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1be16d054bca4d19824b61372c2acb9e" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 41203 } }
I20260620 18:41:12.606142 3936 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1be16d054bca4d19824b61372c2acb9e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:41:12.612622 3943 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1be16d054bca4d19824b61372c2acb9e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1be16d054bca4d19824b61372c2acb9e. Latest consensus state: current_term: 1 leader_uuid: "1be16d054bca4d19824b61372c2acb9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1be16d054bca4d19824b61372c2acb9e" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 41203 } } }
I20260620 18:41:12.613011 3943 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1be16d054bca4d19824b61372c2acb9e [sys.catalog]: This master's current role is: LEADER
I20260620 18:41:12.612249 3942 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1be16d054bca4d19824b61372c2acb9e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1be16d054bca4d19824b61372c2acb9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1be16d054bca4d19824b61372c2acb9e" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 41203 } } }
I20260620 18:41:12.613837 3942 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1be16d054bca4d19824b61372c2acb9e [sys.catalog]: This master's current role is: LEADER
I20260620 18:41:12.627696 3947 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:41:12.634394 3947 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:41:12.635286 3936 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 18:41:12.665843 3947 catalog_manager.cc:1347] Generated new cluster ID: 313e49e0434f452fafa56e26c8defee7
I20260620 18:41:12.665983 3947 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:41:12.678371 3947 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:41:12.680816 3947 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:41:12.691226 3947 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1be16d054bca4d19824b61372c2acb9e: Generated new TSK 0
I20260620 18:41:12.692555 3947 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.115.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.115.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.115.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.1.keytab.
Entry for principal kudu/127.23.115.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.1.keytab.
I20260620 18:41:13.547590 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.115.20:33827/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.115.1:0
--local_ip_for_outbound_sockets=127.23.115.1
--webserver_interface=127.23.115.1
--webserver_port=0
--tserver_master_addrs=127.23.115.62:41203
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.1.keytab
--principal=kudu/127.23.115.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:40323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 18:41:13 dist-test-slave-d645 krb5kdc[3425](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781980873, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 18:41:13.962275 3981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:41:13.962766 3981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:41:13.962970 3981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:41:13.975430 3981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:41:13.975646 3981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:41:13.975873 3981 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.115.1
I20260620 18:41:13.992033 3981 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:40323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.23.115.20:33827/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.1.keytab
--principal=kudu/127.23.115.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.115.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.115.1
--webserver_port=0
--tserver_master_addrs=127.23.115.62:41203
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.115.1
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:41:13.993871 3981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:41:13.995618 3981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:41:14.015375 3991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:14.016672 3992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:14.022449 3994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:41:14.024833 3981 server_base.cc:1034] running on GCE node
I20260620 18:41:14.025669 3981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:41:14.026963 3981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:41:14.028270 3981 hybrid_clock.cc:648] HybridClock initialized: now 1781980874028186 us; error 67 us; skew 500 ppm
I20260620 18:41:14.028746 3981 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:41:14 dist-test-slave-d645 krb5kdc[3425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980874, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:41:14.032986 3981 init.cc:377] Logged in from keytab as kudu/127.23.115.1@KRBTEST.COM (short username kudu)
I20260620 18:41:14.035101 3981 webserver.cc:460] Webserver started at http://127.23.115.1:40193/ using document root <none> and password file <none>
I20260620 18:41:14.036182 3981 fs_manager.cc:359] Metadata directory not provided
I20260620 18:41:14.036361 3981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:41:14.036785 3981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 18:41:14 dist-test-slave-d645 krb5kdc[3425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980874, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.1@KRBTEST.COM for HTTP/127.23.115.20@KRBTEST.COM
I20260620 18:41:14.060281 3981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ts-0/data/instance:
uuid: "71e5f01b9c6c460cb2557dea62904735"
format_stamp: "Formatted at 2026-06-20 18:41:14 on dist-test-slave-d645"
server_key: "3829ba71fa80f286de591dcf3c23e6c9"
server_key_iv: "0df22fb48860b7d277db6ac5d87be8ea"
server_key_version: "kuduclusterkey@0"
I20260620 18:41:14.062353 3981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ts-0/wal/instance:
uuid: "71e5f01b9c6c460cb2557dea62904735"
format_stamp: "Formatted at 2026-06-20 18:41:14 on dist-test-slave-d645"
server_key: "3829ba71fa80f286de591dcf3c23e6c9"
server_key_iv: "0df22fb48860b7d277db6ac5d87be8ea"
server_key_version: "kuduclusterkey@0"
I20260620 18:41:14.070278 3981 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260620 18:41:14.090010 4001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:14.091526 3981 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 18:41:14.091706 3981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ts-0/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ts-0/wal
uuid: "71e5f01b9c6c460cb2557dea62904735"
format_stamp: "Formatted at 2026-06-20 18:41:14 on dist-test-slave-d645"
server_key: "3829ba71fa80f286de591dcf3c23e6c9"
server_key_iv: "0df22fb48860b7d277db6ac5d87be8ea"
server_key_version: "kuduclusterkey@0"
I20260620 18:41:14.091918 3981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:41:14.114741 3981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:41:14.118657 3981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:41:14.118969 3981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:41:14.119997 3981 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:41:14.132598 3981 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:41:14.132755 3981 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260620 18:41:14.132906 3981 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:41:14.132982 3981 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:14.272819 3981 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.1:40059
I20260620 18:41:14.272843 4113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.1:40059 every 8 connection(s)
I20260620 18:41:14.274557 3981 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ts-0/data/info.pb
I20260620 18:41:14.283380 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 3981
I20260620 18:41:14.283615 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ts-0/wal/instance
I20260620 18:41:14.284524 24012 external_mini_cluster.cc:1428] Setting key 1203905bd0aad8acf47337e51609cce3
Jun 20 18:41:14 dist-test-slave-d645 krb5kdc[3425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980873, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.62@KRBTEST.COM for HTTP/127.23.115.20@KRBTEST.COM
Jun 20 18:41:14 dist-test-slave-d645 krb5kdc[3425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980874, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.1@KRBTEST.COM for kudu/127.23.115.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.115.2@KRBTEST.COM; defaulting to no policy
I20260620 18:41:14.332651 4116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:14.277496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.1:41685 (local address 127.23.115.62:41203)
0620 18:41:14.295360 (+ 17864us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:14.295367 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:14.297342 (+ 1975us) server_negotiation.cc:407] Connection header received
0620 18:41:14.299911 (+ 2569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:14.299917 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:14.300046 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:14.300157 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:14.302431 (+ 2274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:14.303626 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:14.317406 (+ 13780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:14.318120 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:14.318244 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:14.322905 (+ 4661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:14.322913 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:14.322918 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:14.322993 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:14.326459 (+ 3466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:14.329791 (+ 3332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:14.329799 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:14.329803 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:14.329997 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:14.330586 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:14.330591 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:14.330595 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:14.331028 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:14.331195 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:14.332012 (+ 817us) server_negotiation.cc:299] Negotiation successful
0620 18:41:14.332245 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17691,"thread_start_us":17646,"threads_started":1}
I20260620 18:41:14.333709 4117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:14.278302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:41203 (local address 127.23.115.1:41685)
0620 18:41:14.296737 (+ 18435us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:14.296800 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:14.299437 (+ 2637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:14.300676 (+ 1239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:14.300690 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:14.301360 (+ 670us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:41:14.302208 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:14.302235 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:14.303914 (+ 1679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:14.303919 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:14.306239 (+ 2320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:14.306267 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:14.318694 (+ 12427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:14.318705 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:14.319125 (+ 420us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:14.319862 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:41:14.319897 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:41:14.322594 (+ 2697us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:41:14.328826 (+ 6232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:14.328838 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:14.328862 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:14.329615 (+ 753us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:14.330170 (+ 555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:14.330175 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:14.330178 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:14.330451 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:14.331284 (+ 833us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:41:14.331290 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:41:14.331633 (+ 343us) client_negotiation.cc:769] Sending connection context
0620 18:41:14.331899 (+ 266us) client_negotiation.cc:240] Negotiation successful
0620 18:41:14.332441 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18090,"thread_start_us":17823,"threads_started":1}
I20260620 18:41:14.335605 4114 heartbeater.cc:347] Connected to a master server at 127.23.115.62:41203
I20260620 18:41:14.335933 4114 heartbeater.cc:464] Registering TS with master...
I20260620 18:41:14.336756 4114 heartbeater.cc:511] Master 127.23.115.62:41203 requested a full tablet report, sending...
I20260620 18:41:14.339373 3901 ts_manager.cc:194] Registered new tserver with Master: 71e5f01b9c6c460cb2557dea62904735 (127.23.115.1:40059)
I20260620 18:41:14.341452 3901 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.115.1@KRBTEST.COM'} at 127.23.115.1:41685
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.115.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.115.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.2.keytab.
Entry for principal kudu/127.23.115.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.2.keytab.
I20260620 18:41:14.378237 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.115.20:33827/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.115.2:0
--local_ip_for_outbound_sockets=127.23.115.2
--webserver_interface=127.23.115.2
--webserver_port=0
--tserver_master_addrs=127.23.115.62:41203
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.2.keytab
--principal=kudu/127.23.115.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:40323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 18:41:14 dist-test-slave-d645 krb5kdc[3425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980873, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.62@KRBTEST.COM for HTTP/127.23.115.20@KRBTEST.COM
W20260620 18:41:14.846100 4120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:41:14.846568 4120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:41:14.846737 4120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:41:14.858924 4120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:41:14.859119 4120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:41:14.859311 4120 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.115.2
I20260620 18:41:14.875219 4120 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:40323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.23.115.20:33827/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.2.keytab
--principal=kudu/127.23.115.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.115.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.115.2
--webserver_port=0
--tserver_master_addrs=127.23.115.62:41203
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.115.2
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:41:14.876957 4120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:41:14.878795 4120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:41:14.893653 4135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:14.893636 4137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:14.894336 4134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:41:14.894897 4120 server_base.cc:1034] running on GCE node
I20260620 18:41:14.895673 4120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:41:14.896869 4120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:41:14.898145 4120 hybrid_clock.cc:648] HybridClock initialized: now 1781980874898057 us; error 84 us; skew 500 ppm
I20260620 18:41:14.898551 4120 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:41:14 dist-test-slave-d645 krb5kdc[3425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980874, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:41:14.902350 4120 init.cc:377] Logged in from keytab as kudu/127.23.115.2@KRBTEST.COM (short username kudu)
I20260620 18:41:14.904150 4120 webserver.cc:460] Webserver started at http://127.23.115.2:37385/ using document root <none> and password file <none>
I20260620 18:41:14.905175 4120 fs_manager.cc:359] Metadata directory not provided
I20260620 18:41:14.905339 4120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:41:14.905720 4120 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 18:41:14 dist-test-slave-d645 krb5kdc[3425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980874, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.2@KRBTEST.COM for HTTP/127.23.115.20@KRBTEST.COM
I20260620 18:41:14.925522 4120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ts-1/data/instance:
uuid: "c68866f8d9924715ab53ec3665024df4"
format_stamp: "Formatted at 2026-06-20 18:41:14 on dist-test-slave-d645"
server_key: "acea6077e4b102fe416f8ddc9ad232ab"
server_key_iv: "c94583a949f3ac729078c6b268db7a37"
server_key_version: "kuduclusterkey@0"
I20260620 18:41:14.927356 4120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ts-1/wal/instance:
uuid: "c68866f8d9924715ab53ec3665024df4"
format_stamp: "Formatted at 2026-06-20 18:41:14 on dist-test-slave-d645"
server_key: "acea6077e4b102fe416f8ddc9ad232ab"
server_key_iv: "c94583a949f3ac729078c6b268db7a37"
server_key_version: "kuduclusterkey@0"
I20260620 18:41:14.936928 4120 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.008s sys 0.000s
I20260620 18:41:14.956614 4144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:14.958539 4120 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260620 18:41:14.958794 4120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ts-1/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ts-1/wal
uuid: "c68866f8d9924715ab53ec3665024df4"
format_stamp: "Formatted at 2026-06-20 18:41:14 on dist-test-slave-d645"
server_key: "acea6077e4b102fe416f8ddc9ad232ab"
server_key_iv: "c94583a949f3ac729078c6b268db7a37"
server_key_version: "kuduclusterkey@0"
I20260620 18:41:14.959169 4120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:41:14.985383 4120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:41:14.990954 4120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:41:14.991334 4120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:41:14.992604 4120 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:41:14.994410 4120 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:41:14.994542 4120 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260620 18:41:14.994699 4120 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:41:14.994787 4120 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:15.051976 4120 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.2:34019
I20260620 18:41:15.052034 4256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.2:34019 every 8 connection(s)
I20260620 18:41:15.053661 4120 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ts-1/data/info.pb
I20260620 18:41:15.061510 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 4120
I20260620 18:41:15.061709 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ts-1/wal/instance
I20260620 18:41:15.062281 24012 external_mini_cluster.cc:1428] Setting key 86c04a5dce9b28d46b45a7f6b0f81881
Jun 20 18:41:15 dist-test-slave-d645 krb5kdc[3425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980874, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.2@KRBTEST.COM for kudu/127.23.115.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.115.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.115.3@KRBTEST.COM" created.
I20260620 18:41:15.106354 4260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:15.057765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.2:35727 (local address 127.23.115.62:41203)
0620 18:41:15.079391 (+ 21626us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:15.079397 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:15.083100 (+ 3703us) server_negotiation.cc:407] Connection header received
0620 18:41:15.083179 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:15.083184 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:15.083295 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:15.083399 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:15.085505 (+ 2106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:15.086625 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:15.089194 (+ 2569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:15.089802 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:15.089918 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:15.094907 (+ 4989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:15.094915 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:15.094920 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:15.095009 (+ 89us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:15.099781 (+ 4772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:15.100795 (+ 1014us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:15.100802 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:15.100806 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:15.100966 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:15.101471 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:15.101476 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:15.101480 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:15.101851 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:15.101981 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:15.103479 (+ 1498us) server_negotiation.cc:299] Negotiation successful
0620 18:41:15.105859 (+ 2380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21472,"thread_start_us":26331,"threads_started":1}
I20260620 18:41:15.109763 4259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:15.057158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:41203 (local address 127.23.115.2:35727)
0620 18:41:15.079989 (+ 22831us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:15.080047 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:15.082529 (+ 2482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:15.083863 (+ 1334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:15.083877 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:15.084479 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:41:15.085290 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:15.085316 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:15.086844 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:15.086849 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:15.088996 (+ 2147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:15.089024 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:15.090336 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:15.090342 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:15.090790 (+ 448us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:15.091713 (+ 923us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:41:15.091745 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:41:15.094609 (+ 2864us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:41:15.099996 (+ 5387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:15.100007 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:15.100030 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:15.100644 (+ 614us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:15.101110 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:15.101114 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:15.101118 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:15.101348 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:15.102079 (+ 731us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:41:15.102088 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:41:15.102785 (+ 697us) client_negotiation.cc:769] Sending connection context
0620 18:41:15.107076 (+ 4291us) client_negotiation.cc:240] Negotiation successful
0620 18:41:15.108248 (+ 1172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22489,"thread_start_us":25768,"threads_started":1}
I20260620 18:41:15.112089 4257 heartbeater.cc:347] Connected to a master server at 127.23.115.62:41203
I20260620 18:41:15.112416 4257 heartbeater.cc:464] Registering TS with master...
I20260620 18:41:15.113302 4257 heartbeater.cc:511] Master 127.23.115.62:41203 requested a full tablet report, sending...
I20260620 18:41:15.115119 3901 ts_manager.cc:194] Registered new tserver with Master: c68866f8d9924715ab53ec3665024df4 (127.23.115.2:34019)
I20260620 18:41:15.116190 3901 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.115.2@KRBTEST.COM'} at 127.23.115.2:35727
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.115.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.3.keytab.
Entry for principal kudu/127.23.115.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.3.keytab.
I20260620 18:41:15.138394 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.115.20:33827/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.115.3:0
--local_ip_for_outbound_sockets=127.23.115.3
--webserver_interface=127.23.115.3
--webserver_port=0
--tserver_master_addrs=127.23.115.62:41203
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.3.keytab
--principal=kudu/127.23.115.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:40323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:41:15.345307 4114 heartbeater.cc:503] Master 127.23.115.62:41203 was elected leader, sending a full tablet report...
W20260620 18:41:15.531183 4263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:41:15.531490 4263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:41:15.531646 4263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:41:15.539367 4263 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:41:15.539490 4263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:41:15.539636 4263 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.115.3
I20260620 18:41:15.549324 4263 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:40323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.23.115.20:33827/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.3.keytab
--principal=kudu/127.23.115.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.115.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.115.3
--webserver_port=0
--tserver_master_addrs=127.23.115.62:41203
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.115.3
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:41:15.550554 4263 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:41:15.551810 4263 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:41:15.564622 4269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:15.564667 4271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:15.564599 4268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:41:15.566200 4263 server_base.cc:1034] running on GCE node
I20260620 18:41:15.566792 4263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:41:15.567833 4263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:41:15.569037 4263 hybrid_clock.cc:648] HybridClock initialized: now 1781980875568991 us; error 41 us; skew 500 ppm
I20260620 18:41:15.569343 4263 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:41:15 dist-test-slave-d645 krb5kdc[3425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980875, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:41:15.572381 4263 init.cc:377] Logged in from keytab as kudu/127.23.115.3@KRBTEST.COM (short username kudu)
I20260620 18:41:15.573900 4263 webserver.cc:460] Webserver started at http://127.23.115.3:37333/ using document root <none> and password file <none>
I20260620 18:41:15.574553 4263 fs_manager.cc:359] Metadata directory not provided
I20260620 18:41:15.574654 4263 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:41:15.574954 4263 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 18:41:15 dist-test-slave-d645 krb5kdc[3425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980875, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.3@KRBTEST.COM for HTTP/127.23.115.20@KRBTEST.COM
I20260620 18:41:15.591270 4263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ts-2/data/instance:
uuid: "25a89431668e4c09ab320a012c9fccc3"
format_stamp: "Formatted at 2026-06-20 18:41:15 on dist-test-slave-d645"
server_key: "2707123097c9ca4ddbe2b6cefd20b255"
server_key_iv: "ef20ab4279f364e2865833d2e57b1a52"
server_key_version: "kuduclusterkey@0"
I20260620 18:41:15.592093 4263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ts-2/wal/instance:
uuid: "25a89431668e4c09ab320a012c9fccc3"
format_stamp: "Formatted at 2026-06-20 18:41:15 on dist-test-slave-d645"
server_key: "2707123097c9ca4ddbe2b6cefd20b255"
server_key_iv: "ef20ab4279f364e2865833d2e57b1a52"
server_key_version: "kuduclusterkey@0"
I20260620 18:41:15.597299 4263 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:41:15.615017 4278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:15.617009 4263 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 18:41:15.617307 4263 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ts-2/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ts-2/wal
uuid: "25a89431668e4c09ab320a012c9fccc3"
format_stamp: "Formatted at 2026-06-20 18:41:15 on dist-test-slave-d645"
server_key: "2707123097c9ca4ddbe2b6cefd20b255"
server_key_iv: "ef20ab4279f364e2865833d2e57b1a52"
server_key_version: "kuduclusterkey@0"
I20260620 18:41:15.617599 4263 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:41:15.641203 4263 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:41:15.646198 4263 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:41:15.646520 4263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:41:15.647544 4263 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:41:15.649495 4263 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:41:15.649624 4263 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:15.649821 4263 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:41:15.649892 4263 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:15.682560 4263 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.3:46047
I20260620 18:41:15.682597 4390 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.3:46047 every 8 connection(s)
I20260620 18:41:15.683789 4263 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ts-2/data/info.pb
I20260620 18:41:15.693323 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 4263
I20260620 18:41:15.693517 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ts-2/wal/instance
I20260620 18:41:15.693981 24012 external_mini_cluster.cc:1428] Setting key 0d2d381abde3e067f1c89ce4d70a987f
Jun 20 18:41:15 dist-test-slave-d645 krb5kdc[3425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980875, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.3@KRBTEST.COM for kudu/127.23.115.62@KRBTEST.COM
I20260620 18:41:15.703436 4393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:15.686259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.3:57323 (local address 127.23.115.62:41203)
0620 18:41:15.686691 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:15.686695 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:15.687551 (+ 856us) server_negotiation.cc:407] Connection header received
0620 18:41:15.689319 (+ 1768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:15.689324 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:15.689434 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:15.689540 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:15.691037 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:15.691760 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:15.693439 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:15.693980 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:15.694107 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:15.698054 (+ 3947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:15.698060 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:15.698064 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:15.698100 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:15.700505 (+ 2405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:15.701211 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:15.701216 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:15.701219 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:15.701341 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:15.701750 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:15.701754 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:15.701756 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:15.702083 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:15.702234 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:15.702960 (+ 726us) server_negotiation.cc:299] Negotiation successful
0620 18:41:15.703141 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":211,"threads_started":1}
I20260620 18:41:15.704352 4394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:15.686735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:41203 (local address 127.23.115.3:57323)
0620 18:41:15.687401 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:15.687442 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:15.689062 (+ 1620us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:15.689770 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:15.689779 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:15.690180 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:41:15.690842 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:15.690861 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:15.691899 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:15.691903 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:15.693249 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:15.693273 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:15.694216 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:15.694224 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:15.694502 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:15.695220 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:41:15.695249 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:41:15.697795 (+ 2546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:41:15.700637 (+ 2842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:15.700645 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:15.700657 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:15.701095 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:15.701419 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:15.701423 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:15.701425 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:15.701608 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:15.702232 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:41:15.702239 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:41:15.702567 (+ 328us) client_negotiation.cc:769] Sending connection context
0620 18:41:15.702807 (+ 240us) client_negotiation.cc:240] Negotiation successful
0620 18:41:15.703233 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":276,"threads_started":1}
I20260620 18:41:15.705866 4391 heartbeater.cc:347] Connected to a master server at 127.23.115.62:41203
I20260620 18:41:15.706228 4391 heartbeater.cc:464] Registering TS with master...
I20260620 18:41:15.706964 4391 heartbeater.cc:511] Master 127.23.115.62:41203 requested a full tablet report, sending...
I20260620 18:41:15.708662 3900 ts_manager.cc:194] Registered new tserver with Master: 25a89431668e4c09ab320a012c9fccc3 (127.23.115.3:46047)
I20260620 18:41:15.709553 24012 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:41:15.709785 3900 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.115.3@KRBTEST.COM'} at 127.23.115.3:57323
I20260620 18:41:15.712070 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 3981
Jun 20 18:41:15 dist-test-slave-d645 krb5kdc[3425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980871, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.115.1@KRBTEST.COM
I20260620 18:41:15.730623 4396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:15.713342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54400 (local address 127.23.115.1:40059)
0620 18:41:15.713905 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:15.713911 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:15.713956 (+ 45us) server_negotiation.cc:407] Connection header received
0620 18:41:15.714110 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:15.714115 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:15.714307 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:15.714430 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:15.715815 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:15.716803 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:15.718219 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:15.718775 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:15.718879 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:15.723825 (+ 4946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:15.723839 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:15.723853 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:15.723895 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:15.726547 (+ 2652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:15.727365 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:15.727373 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:15.727375 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:15.727543 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:15.728327 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:15.728333 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:15.728336 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:15.728985 (+ 649us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:15.729406 (+ 421us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:15.729877 (+ 471us) server_negotiation.cc:299] Negotiation successful
0620 18:41:15.730192 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":268,"threads_started":1}
I20260620 18:41:15.731980 4109 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:41:15.814672 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 3981
I20260620 18:41:15.828210 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 4120
Jun 20 18:41:15 dist-test-slave-d645 krb5kdc[3425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980871, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.115.2@KRBTEST.COM
I20260620 18:41:15.845279 4398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:15.829418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32980 (local address 127.23.115.2:34019)
0620 18:41:15.830023 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:15.830030 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:15.830058 (+ 28us) server_negotiation.cc:407] Connection header received
0620 18:41:15.830117 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:15.830123 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:15.830326 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:15.830453 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:15.831745 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:15.832755 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:15.834321 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:15.834872 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:15.835041 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:15.839200 (+ 4159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:15.839212 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:15.839227 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:15.839262 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:15.841930 (+ 2668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:15.842651 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:15.842658 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:15.842661 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:15.842896 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:15.843249 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:15.843252 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:15.843254 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:15.843627 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:15.843877 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:15.844539 (+ 662us) server_negotiation.cc:299] Negotiation successful
0620 18:41:15.844842 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":346,"threads_started":1}
I20260620 18:41:15.846822 4252 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:41:15.930425 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 4120
I20260620 18:41:15.942639 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 4263
Jun 20 18:41:15 dist-test-slave-d645 krb5kdc[3425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980871, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.115.3@KRBTEST.COM
I20260620 18:41:15.959342 4400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:15.943654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59386 (local address 127.23.115.3:46047)
0620 18:41:15.944319 (+ 665us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:15.944325 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:15.944352 (+ 27us) server_negotiation.cc:407] Connection header received
0620 18:41:15.944409 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:15.944415 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:15.944663 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:15.944803 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:15.946204 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:15.947277 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:15.948542 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:15.949122 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:15.949229 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:15.953796 (+ 4567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:15.953807 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:15.953822 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:15.953862 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:15.956652 (+ 2790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:15.957300 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:15.957307 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:15.957310 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:15.957435 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:15.957778 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:15.957783 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:15.957785 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:15.958117 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:15.958395 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:15.958704 (+ 309us) server_negotiation.cc:299] Negotiation successful
0620 18:41:15.958993 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":343,"threads_started":1}
I20260620 18:41:15.960603 4386 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:41:16.046492 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 4263
I20260620 18:41:16.060758 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 3869
I20260620 18:41:16.061995 3931 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:41:16.167243 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 3869
I20260620 18:41:16.189244 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.115.62:41203
--webserver_interface=127.23.115.62
--webserver_port=44517
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab
--principal=kudu/127.23.115.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:40323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.115.62:41203
--ranger_config_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.115.20:33827/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.23.115.20:33827/kms with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:41:16.507791 4403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:41:16.508087 4403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:41:16.508175 4403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:41:16.515412 4403 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:41:16.515527 4403 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:41:16.515574 4403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:41:16.515630 4403 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:41:16.515674 4403 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:41:16.525842 4403 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:40323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.23.115.20:33827/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.23.115.62:41203
--ranger_config_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab
--principal=kudu/127.23.115.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.115.62:41203
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.115.62
--webserver_port=44517
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:41:16.527099 4403 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:41:16.528373 4403 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:41:16.540366 4408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:16.540349 4411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:16.540568 4409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:41:16.544739 4403 server_base.cc:1034] running on GCE node
I20260620 18:41:16.545565 4403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:41:16.547065 4403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:41:16.548286 4403 hybrid_clock.cc:648] HybridClock initialized: now 1781980876548232 us; error 49 us; skew 500 ppm
I20260620 18:41:16.548645 4403 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:41:16 dist-test-slave-d645 krb5kdc[3425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980876, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:41:16.552161 4403 init.cc:377] Logged in from keytab as kudu/127.23.115.62@KRBTEST.COM (short username kudu)
I20260620 18:41:16.553767 4403 webserver.cc:460] Webserver started at http://127.23.115.62:44517/ using document root <none> and password file <none>
I20260620 18:41:16.554430 4403 fs_manager.cc:359] Metadata directory not provided
I20260620 18:41:16.554548 4403 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 18:41:16 dist-test-slave-d645 krb5kdc[3425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980876, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.62@KRBTEST.COM for HTTP/127.23.115.20@KRBTEST.COM
I20260620 18:41:16.642805 4403 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.006s sys 0.000s
I20260620 18:41:16.646451 4419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:16.648036 4403 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 18:41:16.648212 4403 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/master-0/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/master-0/wal
uuid: "1be16d054bca4d19824b61372c2acb9e"
format_stamp: "Formatted at 2026-06-20 18:41:12 on dist-test-slave-d645"
server_key: "96a535bc1032c2a7ba375b3554f46afe"
server_key_iv: "5e328a4c2cad679b700455936db9651e"
server_key_version: "kuduclusterkey@0"
I20260620 18:41:16.649173 4403 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:41:16.667480 4403 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:41:16.672621 4403 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:41:16.673139 4403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:41:16.693359 4403 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.62:41203
I20260620 18:41:16.693393 4470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.62:41203 every 8 connection(s)
I20260620 18:41:16.694808 4403 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
I20260620 18:41:16.701607 4471 sys_catalog.cc:263] Verifying existing consensus state
I20260620 18:41:16.702598 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 4403
I20260620 18:41:16.703477 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.115.20:33827/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.115.1:40059
--local_ip_for_outbound_sockets=127.23.115.1
--tserver_master_addrs=127.23.115.62:41203
--webserver_port=40193
--webserver_interface=127.23.115.1
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.1.keytab
--principal=kudu/127.23.115.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:40323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.23.115.20:33827/kms with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:41:16.705462 4471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1be16d054bca4d19824b61372c2acb9e: Bootstrap starting.
I20260620 18:41:16.726574 4471 log.cc:824] T 00000000000000000000000000000000 P 1be16d054bca4d19824b61372c2acb9e: Log is configured to *not* fsync() on all Append() calls
I20260620 18:41:16.737507 4471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1be16d054bca4d19824b61372c2acb9e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 18:41:16.738083 4471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1be16d054bca4d19824b61372c2acb9e: Bootstrap complete.
I20260620 18:41:16.748277 4471 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1be16d054bca4d19824b61372c2acb9e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1be16d054bca4d19824b61372c2acb9e" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 41203 } }
I20260620 18:41:16.749150 4471 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1be16d054bca4d19824b61372c2acb9e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1be16d054bca4d19824b61372c2acb9e, State: Initialized, Role: FOLLOWER
I20260620 18:41:16.749782 4471 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1be16d054bca4d19824b61372c2acb9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1be16d054bca4d19824b61372c2acb9e" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 41203 } }
I20260620 18:41:16.750013 4471 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1be16d054bca4d19824b61372c2acb9e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:41:16.750129 4471 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1be16d054bca4d19824b61372c2acb9e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:41:16.750278 4471 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1be16d054bca4d19824b61372c2acb9e [term 1 FOLLOWER]: Advancing to term 2
I20260620 18:41:16.754233 4471 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1be16d054bca4d19824b61372c2acb9e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1be16d054bca4d19824b61372c2acb9e" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 41203 } }
I20260620 18:41:16.754922 4471 leader_election.cc:302] T 00000000000000000000000000000000 P 1be16d054bca4d19824b61372c2acb9e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1be16d054bca4d19824b61372c2acb9e; no voters:
I20260620 18:41:16.755831 4471 leader_election.cc:288] T 00000000000000000000000000000000 P 1be16d054bca4d19824b61372c2acb9e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 18:41:16.755980 4475 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1be16d054bca4d19824b61372c2acb9e [term 2 FOLLOWER]: Leader election won for term 2
I20260620 18:41:16.758111 4471 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1be16d054bca4d19824b61372c2acb9e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:41:16.762959 4475 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1be16d054bca4d19824b61372c2acb9e [term 2 LEADER]: Becoming Leader. State: Replica: 1be16d054bca4d19824b61372c2acb9e, State: Running, Role: LEADER
I20260620 18:41:16.763761 4475 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1be16d054bca4d19824b61372c2acb9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1be16d054bca4d19824b61372c2acb9e" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 41203 } }
I20260620 18:41:16.766373 4476 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1be16d054bca4d19824b61372c2acb9e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "1be16d054bca4d19824b61372c2acb9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1be16d054bca4d19824b61372c2acb9e" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 41203 } } }
I20260620 18:41:16.766804 4476 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1be16d054bca4d19824b61372c2acb9e [sys.catalog]: This master's current role is: LEADER
I20260620 18:41:16.767336 4477 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1be16d054bca4d19824b61372c2acb9e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1be16d054bca4d19824b61372c2acb9e. Latest consensus state: current_term: 2 leader_uuid: "1be16d054bca4d19824b61372c2acb9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1be16d054bca4d19824b61372c2acb9e" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 41203 } } }
I20260620 18:41:16.767622 4477 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1be16d054bca4d19824b61372c2acb9e [sys.catalog]: This master's current role is: LEADER
I20260620 18:41:16.769531 4484 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:41:16.783772 4484 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:41:16.785421 4484 catalog_manager.cc:1259] Loaded cluster ID: 313e49e0434f452fafa56e26c8defee7
I20260620 18:41:16.785604 4484 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:41:16.789827 4484 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:41:16.791515 4484 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1be16d054bca4d19824b61372c2acb9e: Loaded TSK: 0
I20260620 18:41:16.793260 4484 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:41:16.800791 4471 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260620 18:41:17.158022 4473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:41:17.158540 4473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:41:17.158741 4473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:41:17.170570 4473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:41:17.170801 4473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:41:17.171087 4473 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.115.1
I20260620 18:41:17.187670 4473 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:40323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.23.115.20:33827/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.1.keytab
--principal=kudu/127.23.115.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.115.1:40059
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.115.1
--webserver_port=40193
--tserver_master_addrs=127.23.115.62:41203
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.115.1
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:41:17.189654 4473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:41:17.191526 4473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:41:17.209862 4509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:17.210502 4510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:41:17.222625 4473 server_base.cc:1034] running on GCE node
W20260620 18:41:17.223397 4512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:41:17.224252 4473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:41:17.225575 4473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:41:17.226876 4473 hybrid_clock.cc:648] HybridClock initialized: now 1781980877226797 us; error 63 us; skew 500 ppm
I20260620 18:41:17.227327 4473 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:41:17 dist-test-slave-d645 krb5kdc[3425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980877, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:41:17.231488 4473 init.cc:377] Logged in from keytab as kudu/127.23.115.1@KRBTEST.COM (short username kudu)
I20260620 18:41:17.233631 4473 webserver.cc:460] Webserver started at http://127.23.115.1:40193/ using document root <none> and password file <none>
I20260620 18:41:17.234606 4473 fs_manager.cc:359] Metadata directory not provided
I20260620 18:41:17.234798 4473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 18:41:17 dist-test-slave-d645 krb5kdc[3425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980877, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.1@KRBTEST.COM for HTTP/127.23.115.20@KRBTEST.COM
I20260620 18:41:17.285018 4473 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.003s sys 0.004s
I20260620 18:41:17.295408 4520 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:17.297515 4473 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.005s sys 0.005s
I20260620 18:41:17.297791 4473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ts-0/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ts-0/wal
uuid: "71e5f01b9c6c460cb2557dea62904735"
format_stamp: "Formatted at 2026-06-20 18:41:14 on dist-test-slave-d645"
server_key: "3829ba71fa80f286de591dcf3c23e6c9"
server_key_iv: "0df22fb48860b7d277db6ac5d87be8ea"
server_key_version: "kuduclusterkey@0"
I20260620 18:41:17.299362 4473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:41:17.322003 4473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:41:17.327491 4473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:41:17.327881 4473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:41:17.331161 4473 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:41:17.333209 4473 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:41:17.333380 4473 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260620 18:41:17.333577 4473 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:41:17.333739 4473 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:17.401014 4473 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.1:40059
I20260620 18:41:17.401103 4633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.1:40059 every 8 connection(s)
I20260620 18:41:17.402565 4473 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ts-0/data/info.pb
I20260620 18:41:17.410741 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 4473
I20260620 18:41:17.411600 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.115.20:33827/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.115.2:34019
--local_ip_for_outbound_sockets=127.23.115.2
--tserver_master_addrs=127.23.115.62:41203
--webserver_port=37385
--webserver_interface=127.23.115.2
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.2.keytab
--principal=kudu/127.23.115.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:40323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.23.115.20:33827/kms with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 18:41:17 dist-test-slave-d645 krb5kdc[3425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980877, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.1@KRBTEST.COM for kudu/127.23.115.62@KRBTEST.COM
I20260620 18:41:17.434805 4637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:17.405760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:41203 (local address 127.23.115.1:32775)
0620 18:41:17.406670 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:17.406706 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:17.408645 (+ 1939us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:17.410272 (+ 1627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:17.410283 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:17.410795 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:41:17.411437 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:17.411454 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:17.413197 (+ 1743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:17.413202 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:17.414508 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:17.414530 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:17.420176 (+ 5646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:17.420186 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:17.420491 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:17.423717 (+ 3226us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:41:17.423753 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:41:17.426419 (+ 2666us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:41:17.429982 (+ 3563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:17.429992 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:17.430008 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:17.430439 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:17.430957 (+ 518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:17.430962 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:17.430966 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:17.431245 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:17.432251 (+ 1006us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:41:17.432259 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:41:17.432624 (+ 365us) client_negotiation.cc:769] Sending connection context
0620 18:41:17.432884 (+ 260us) client_negotiation.cc:240] Negotiation successful
0620 18:41:17.433474 (+ 590us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":696,"thread_start_us":502,"threads_started":1}
I20260620 18:41:17.436612 4636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:17.405658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.1:32775 (local address 127.23.115.62:41203)
0620 18:41:17.406371 (+ 713us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:17.406390 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:17.406823 (+ 433us) server_negotiation.cc:407] Connection header received
0620 18:41:17.409276 (+ 2453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:17.409293 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:17.409736 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:17.410082 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:17.411635 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:17.413004 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:17.414658 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:17.419515 (+ 4857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:17.419737 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:17.426748 (+ 7011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:17.426757 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:17.426773 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:17.426818 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:17.429808 (+ 2990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:17.430597 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:17.430604 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:17.430609 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:17.430771 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:17.431393 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:17.431398 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:17.431402 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:17.432109 (+ 707us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:17.432299 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:17.435260 (+ 2961us) server_negotiation.cc:299] Negotiation successful
0620 18:41:17.435644 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":294,"threads_started":1}
I20260620 18:41:17.439831 4634 heartbeater.cc:347] Connected to a master server at 127.23.115.62:41203
I20260620 18:41:17.440145 4634 heartbeater.cc:464] Registering TS with master...
I20260620 18:41:17.441035 4634 heartbeater.cc:511] Master 127.23.115.62:41203 requested a full tablet report, sending...
W20260620 18:41:17.444012 4634 heartbeater.cc:644] Failed to heartbeat to 127.23.115.62:41203 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 18:41:17.444244 4634 heartbeater.cc:464] Registering TS with master...
I20260620 18:41:17.444566 4634 heartbeater.cc:511] Master 127.23.115.62:41203 requested a full tablet report, sending...
I20260620 18:41:17.445999 4634 heartbeater.cc:464] Registering TS with master...
I20260620 18:41:17.446283 4634 heartbeater.cc:511] Master 127.23.115.62:41203 requested a full tablet report, sending...
W20260620 18:41:17.447476 4634 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 18:41:17.832086 4638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:41:17.832398 4638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:41:17.832605 4638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:41:17.841897 4638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:41:17.842011 4638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:41:17.842159 4638 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.115.2
I20260620 18:41:17.853996 4638 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:40323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.23.115.20:33827/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.2.keytab
--principal=kudu/127.23.115.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.115.2:34019
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.115.2
--webserver_port=37385
--tserver_master_addrs=127.23.115.62:41203
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.115.2
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:41:17.855702 4638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:41:17.857303 4638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:41:17.881712 4653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:17.883101 4652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:17.883760 4655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:41:17.884719 4638 server_base.cc:1034] running on GCE node
I20260620 18:41:17.885352 4638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:41:17.886415 4638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:41:17.887759 4638 hybrid_clock.cc:648] HybridClock initialized: now 1781980877887665 us; error 39 us; skew 500 ppm
I20260620 18:41:17.888334 4638 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:41:17 dist-test-slave-d645 krb5kdc[3425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980877, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:41:17.892135 4638 init.cc:377] Logged in from keytab as kudu/127.23.115.2@KRBTEST.COM (short username kudu)
I20260620 18:41:17.894129 4638 webserver.cc:460] Webserver started at http://127.23.115.2:37385/ using document root <none> and password file <none>
I20260620 18:41:17.895134 4638 fs_manager.cc:359] Metadata directory not provided
I20260620 18:41:17.895324 4638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 18:41:17 dist-test-slave-d645 krb5kdc[3425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980877, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.2@KRBTEST.COM for HTTP/127.23.115.20@KRBTEST.COM
I20260620 18:41:17.924443 4638 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.004s sys 0.000s
I20260620 18:41:17.938875 4663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:17.948946 4638 fs_manager.cc:725] Time spent opening block manager: real 0.022s user 0.005s sys 0.009s
I20260620 18:41:17.949165 4638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ts-1/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ts-1/wal
uuid: "c68866f8d9924715ab53ec3665024df4"
format_stamp: "Formatted at 2026-06-20 18:41:14 on dist-test-slave-d645"
server_key: "acea6077e4b102fe416f8ddc9ad232ab"
server_key_iv: "c94583a949f3ac729078c6b268db7a37"
server_key_version: "kuduclusterkey@0"
I20260620 18:41:17.950649 4638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:41:18.005767 4638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:41:18.012147 4638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:41:18.012506 4638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:41:18.014050 4638 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:41:18.015790 4638 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:41:18.015893 4638 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:18.016012 4638 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:41:18.016090 4638 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 18:41:18 dist-test-slave-d645 krb5kdc[3425](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781980878, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:41:18.075851 4638 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.2:34019
I20260620 18:41:18.076349 4777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.2:34019 every 8 connection(s)
I20260620 18:41:18.077944 4638 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ts-1/data/info.pb
I20260620 18:41:18.088470 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 4638
I20260620 18:41:18.089244 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.115.20:33827/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.115.3:46047
--local_ip_for_outbound_sockets=127.23.115.3
--tserver_master_addrs=127.23.115.62:41203
--webserver_port=37333
--webserver_interface=127.23.115.3
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.3.keytab
--principal=kudu/127.23.115.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:40323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.23.115.20:33827/kms with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 18:41:18 dist-test-slave-d645 krb5kdc[3425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980877, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.2@KRBTEST.COM for kudu/127.23.115.62@KRBTEST.COM
I20260620 18:41:18.149137 4781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:18.098912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:41203 (local address 127.23.115.2:33501)
0620 18:41:18.123615 (+ 24703us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:18.123691 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:18.126425 (+ 2734us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:18.128557 (+ 2132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:18.128570 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:18.129237 (+ 667us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:41:18.130075 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:18.130103 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:18.131795 (+ 1692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:18.131801 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:18.133861 (+ 2060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:18.133881 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:18.135381 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:18.135391 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:18.135716 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:18.136529 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:41:18.136568 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:41:18.139512 (+ 2944us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:41:18.143893 (+ 4381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:18.143904 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:18.143926 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:18.144605 (+ 679us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:18.145105 (+ 500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:18.145110 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:18.145115 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:18.145402 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:18.146195 (+ 793us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:41:18.146205 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:41:18.146678 (+ 473us) client_negotiation.cc:769] Sending connection context
0620 18:41:18.147031 (+ 353us) client_negotiation.cc:240] Negotiation successful
0620 18:41:18.147638 (+ 607us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24295,"thread_start_us":27861,"threads_started":1}
I20260620 18:41:18.150355 4782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:18.099540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.2:33501 (local address 127.23.115.62:41203)
0620 18:41:18.122797 (+ 23257us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:18.122804 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:18.127779 (+ 4975us) server_negotiation.cc:407] Connection header received
0620 18:41:18.127868 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:18.127873 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:18.127974 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:18.128111 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:18.130310 (+ 2199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:18.131534 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:18.134110 (+ 2576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:18.134824 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:18.135012 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:18.139841 (+ 4829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:18.139849 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:18.139855 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:18.139904 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:18.143683 (+ 3779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:18.144790 (+ 1107us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:18.144797 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:18.144801 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:18.144962 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:18.145552 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:18.145558 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:18.145562 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:18.145955 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:18.146105 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:18.149665 (+ 3560us) server_negotiation.cc:299] Negotiation successful
0620 18:41:18.149932 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23097,"thread_start_us":23293,"threads_started":1}
I20260620 18:41:18.152719 4778 heartbeater.cc:347] Connected to a master server at 127.23.115.62:41203
I20260620 18:41:18.153075 4778 heartbeater.cc:464] Registering TS with master...
I20260620 18:41:18.154070 4778 heartbeater.cc:511] Master 127.23.115.62:41203 requested a full tablet report, sending...
I20260620 18:41:18.156553 4436 ts_manager.cc:194] Registered new tserver with Master: c68866f8d9924715ab53ec3665024df4 (127.23.115.2:34019)
I20260620 18:41:18.158763 4436 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.115.2@KRBTEST.COM'} at 127.23.115.2:33501
I20260620 18:41:18.447638 4634 heartbeater.cc:464] Registering TS with master...
I20260620 18:41:18.448088 4634 heartbeater.cc:511] Master 127.23.115.62:41203 requested a full tablet report, sending...
I20260620 18:41:18.449450 4436 ts_manager.cc:194] Registered new tserver with Master: 71e5f01b9c6c460cb2557dea62904735 (127.23.115.1:40059)
I20260620 18:41:18.450609 4436 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.115.1@KRBTEST.COM'} at 127.23.115.1:32775
Jun 20 18:41:18 dist-test-slave-d645 krb5kdc[3425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980878, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.62@KRBTEST.COM for HTTP/127.23.115.20@KRBTEST.COM
W20260620 18:41:18.586370 4780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:41:18.586830 4780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:41:18.587018 4780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:41:18.597481 4780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:41:18.597703 4780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:41:18.597916 4780 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.115.3
I20260620 18:41:18.612888 4780 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:40323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.23.115.20:33827/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.3.keytab
--principal=kudu/127.23.115.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.115.3:46047
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.115.3
--webserver_port=37333
--tserver_master_addrs=127.23.115.62:41203
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.115.3
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:41:18.614749 4780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:41:18.616561 4780 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:41:18.634341 4788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:18.634356 4789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:18.642168 4791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:41:18.643157 4780 server_base.cc:1034] running on GCE node
I20260620 18:41:18.643807 4780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:41:18.644855 4780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:41:18.646083 4780 hybrid_clock.cc:648] HybridClock initialized: now 1781980878646025 us; error 45 us; skew 500 ppm
I20260620 18:41:18.646417 4780 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:41:18 dist-test-slave-d645 krb5kdc[3425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980878, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:41:18.649467 4780 init.cc:377] Logged in from keytab as kudu/127.23.115.3@KRBTEST.COM (short username kudu)
I20260620 18:41:18.655102 4780 webserver.cc:460] Webserver started at http://127.23.115.3:37333/ using document root <none> and password file <none>
I20260620 18:41:18.656044 4780 fs_manager.cc:359] Metadata directory not provided
I20260620 18:41:18.656186 4780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 18:41:18 dist-test-slave-d645 krb5kdc[3425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980878, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.3@KRBTEST.COM for HTTP/127.23.115.20@KRBTEST.COM
I20260620 18:41:18.676645 4780 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 18:41:18.679396 4799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:18.680962 4780 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 18:41:18.681120 4780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ts-2/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ts-2/wal
uuid: "25a89431668e4c09ab320a012c9fccc3"
format_stamp: "Formatted at 2026-06-20 18:41:15 on dist-test-slave-d645"
server_key: "2707123097c9ca4ddbe2b6cefd20b255"
server_key_iv: "ef20ab4279f364e2865833d2e57b1a52"
server_key_version: "kuduclusterkey@0"
I20260620 18:41:18.682070 4780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:41:18.710390 4780 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:41:18.715746 4780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:41:18.716074 4780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:41:18.717355 4780 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:41:18.719115 4780 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:41:18.719236 4780 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:18.719352 4780 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:41:18.719430 4780 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 18:41:18 dist-test-slave-d645 krb5kdc[3425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980878, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.62@KRBTEST.COM for HTTP/127.23.115.20@KRBTEST.COM
I20260620 18:41:18.825505 4780 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.3:46047
I20260620 18:41:18.825562 4912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.3:46047 every 8 connection(s)
I20260620 18:41:18.827526 4780 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ts-2/data/info.pb
I20260620 18:41:18.831660 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 4780
I20260620 18:41:18.916834 4915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:18.833028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58698 (local address 127.23.115.62:41203)
0620 18:41:18.859563 (+ 26535us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:18.859570 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:18.860401 (+ 831us) server_negotiation.cc:407] Connection header received
0620 18:41:18.860474 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:18.860480 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:18.860576 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:18.860670 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:18.870737 (+ 10067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:18.871851 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:18.893543 (+ 21692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:18.894250 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:18.894386 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:18.909531 (+ 15145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:18.909539 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:18.909545 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:18.909599 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:18.913176 (+ 3577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:18.914205 (+ 1029us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:18.914210 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:18.914214 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:18.914362 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:18.914816 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:18.914821 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:18.914824 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:18.915209 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:18.915456 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:18.916032 (+ 576us) server_negotiation.cc:299] Negotiation successful
0620 18:41:18.916421 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26376,"thread_start_us":26541,"threads_started":1}
Jun 20 18:41:18 dist-test-slave-d645 krb5kdc[3425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980878, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.3@KRBTEST.COM for kudu/127.23.115.62@KRBTEST.COM
I20260620 18:41:18.954607 4915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:18.894672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.3:40115 (local address 127.23.115.62:41203)
0620 18:41:18.917388 (+ 22716us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:18.917394 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:18.922777 (+ 5383us) server_negotiation.cc:407] Connection header received
0620 18:41:18.932360 (+ 9583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:18.932367 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:18.932465 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:18.932567 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:18.937353 (+ 4786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:18.938427 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:18.941173 (+ 2746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:18.941776 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:18.941866 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:18.946434 (+ 4568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:18.946442 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:18.946448 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:18.946492 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:18.950570 (+ 4078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:18.951601 (+ 1031us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:18.951607 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:18.951611 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:18.951759 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:18.952328 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:18.952334 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:18.952337 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:18.952703 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:18.952833 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:18.953965 (+ 1132us) server_negotiation.cc:299] Negotiation successful
0620 18:41:18.954238 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22570,"thread_start_us":26595,"threads_started":1}
I20260620 18:41:18.957237 4917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:18.893000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:41203 (local address 127.23.115.3:40115)
0620 18:41:18.922370 (+ 29370us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:18.922451 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:18.930822 (+ 8371us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:18.933144 (+ 2322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:18.933157 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:18.933788 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:41:18.937139 (+ 3351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:18.937173 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:18.938652 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:18.938657 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:18.941007 (+ 2350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:18.941024 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:18.942221 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:18.942228 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:18.942505 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:18.943300 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:41:18.943332 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:41:18.946140 (+ 2808us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:41:18.950752 (+ 4612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:18.950763 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:18.950784 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:18.951456 (+ 672us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:18.951904 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:18.951909 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:18.951912 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:18.952194 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:18.952943 (+ 749us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:41:18.952952 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:41:18.953497 (+ 545us) client_negotiation.cc:769] Sending connection context
0620 18:41:18.955273 (+ 1776us) client_negotiation.cc:240] Negotiation successful
0620 18:41:18.955912 (+ 639us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28986,"thread_start_us":37841,"threads_started":1}
I20260620 18:41:18.958961 4913 heartbeater.cc:347] Connected to a master server at 127.23.115.62:41203
I20260620 18:41:18.959267 4913 heartbeater.cc:464] Registering TS with master...
I20260620 18:41:18.960058 4913 heartbeater.cc:511] Master 127.23.115.62:41203 requested a full tablet report, sending...
I20260620 18:41:18.963083 4436 ts_manager.cc:194] Registered new tserver with Master: 25a89431668e4c09ab320a012c9fccc3 (127.23.115.3:46047)
I20260620 18:41:18.964133 4436 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.115.3@KRBTEST.COM'} at 127.23.115.3:40115
I20260620 18:41:18.975559 24012 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:41:18.996855 4915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:18.979895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58708 (local address 127.23.115.62:41203)
0620 18:41:18.980066 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:18.980071 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:18.980585 (+ 514us) server_negotiation.cc:407] Connection header received
0620 18:41:18.980861 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:18.980867 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:18.980947 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:18.981016 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:18.982749 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:18.983927 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:18.986075 (+ 2148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:18.986695 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:18.986773 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:18.989577 (+ 2804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:18.989584 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:18.989589 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:18.989634 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:18.993386 (+ 3752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:18.994300 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:18.994305 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:18.994309 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:18.994471 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:18.994959 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:18.994964 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:18.994968 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:18.995358 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:18.995608 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:18.996224 (+ 616us) server_negotiation.cc:299] Negotiation successful
0620 18:41:18.996476 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 18:41:19.003396 4436 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58708:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:41:19.014880 4436 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:41:19.073923 4930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:19.048927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.2:34019 (local address 127.0.0.1:51596)
0620 18:41:19.056592 (+ 7665us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:19.056615 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:19.056719 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:19.058149 (+ 1430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:19.058158 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:19.058204 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:41:19.058587 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:19.058601 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:19.060526 (+ 1925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:19.060532 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:19.062800 (+ 2268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:19.062816 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:19.072690 (+ 9874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:19.072698 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:19.073025 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:19.073106 (+ 81us) client_negotiation.cc:769] Sending connection context
0620 18:41:19.073235 (+ 129us) client_negotiation.cc:240] Negotiation successful
0620 18:41:19.073470 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7500,"thread_start_us":7466,"threads_started":1}
I20260620 18:41:19.083613 4931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:19.049463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51596 (local address 127.23.115.2:34019)
0620 18:41:19.057203 (+ 7740us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:19.057213 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:19.057245 (+ 32us) server_negotiation.cc:407] Connection header received
0620 18:41:19.057356 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:19.057365 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:19.057663 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:19.057874 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:41:19.058794 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:19.060312 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:19.068199 (+ 7887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:19.072382 (+ 4183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:19.082583 (+ 10201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:19.082853 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:19.082985 (+ 132us) server_negotiation.cc:299] Negotiation successful
0620 18:41:19.083208 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7528,"thread_start_us":8001,"threads_started":1}
I20260620 18:41:19.087066 4930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:19.058070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.1:40059 (local address 127.0.0.1:58710)
0620 18:41:19.074509 (+ 16439us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:19.074531 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:19.074654 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:19.075717 (+ 1063us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:19.075723 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:19.075753 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:41:19.075941 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:19.075953 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:19.078073 (+ 2120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:19.078079 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:19.080125 (+ 2046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:19.080143 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:19.084735 (+ 4592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:19.084743 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:19.086245 (+ 1502us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:19.086289 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 18:41:19.086408 (+ 119us) client_negotiation.cc:240] Negotiation successful
0620 18:41:19.086593 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16328,"mutex_wait_us":1246,"thread_start_us":19147,"threads_started":1}
I20260620 18:41:19.089362 4713 tablet_service.cc:1467] Processing CreateTablet for tablet b1f5b8ea4adc443090c7cec266f2fad4 (DEFAULT_TABLE table=test-table [id=c1ec1c3ea9be4a819f47aff93259a941]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:41:19.091080 4713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1f5b8ea4adc443090c7cec266f2fad4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:41:19.091166 4928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:19.045199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.3:46047 (local address 127.0.0.1:55952)
0620 18:41:19.055631 (+ 10432us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:19.055725 (+ 94us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:19.055979 (+ 254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:19.068960 (+ 12981us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:19.068967 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:19.069003 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:41:19.069471 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:19.069487 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:19.084008 (+ 14521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:19.084016 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:19.087901 (+ 3885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:19.087919 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:19.090228 (+ 2309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:19.090236 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:19.090421 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:19.090461 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 18:41:19.090571 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 18:41:19.090720 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10090,"mutex_wait_us":1,"thread_start_us":10420,"threads_started":1}
I20260620 18:41:19.092152 4929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:19.045602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55952 (local address 127.23.115.3:46047)
0620 18:41:19.067477 (+ 21875us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:19.067486 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:19.067528 (+ 42us) server_negotiation.cc:407] Connection header received
0620 18:41:19.067643 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:19.067652 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:19.067918 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:19.068967 (+ 1049us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:41:19.069725 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:19.071329 (+ 1604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:19.088139 (+ 16810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:19.089660 (+ 1521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:19.089805 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:19.090063 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:19.091590 (+ 1527us) server_negotiation.cc:299] Negotiation successful
0620 18:41:19.091775 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21661,"thread_start_us":22539,"threads_started":1}
I20260620 18:41:19.096499 4933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:19.058650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58710 (local address 127.23.115.1:40059)
0620 18:41:19.069729 (+ 11079us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:19.069738 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:19.074791 (+ 5053us) server_negotiation.cc:407] Connection header received
0620 18:41:19.074912 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:19.074922 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:19.075222 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:19.075445 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:41:19.076148 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:19.077866 (+ 1718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:19.080358 (+ 2492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:19.081995 (+ 1637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:19.082140 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:19.082417 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:19.095939 (+ 13522us) server_negotiation.cc:299] Negotiation successful
0620 18:41:19.096165 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10876,"thread_start_us":11412,"threads_started":1}
I20260620 18:41:19.096233 4847 tablet_service.cc:1467] Processing CreateTablet for tablet b1f5b8ea4adc443090c7cec266f2fad4 (DEFAULT_TABLE table=test-table [id=c1ec1c3ea9be4a819f47aff93259a941]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:41:19.097764 4847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1f5b8ea4adc443090c7cec266f2fad4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:41:19.100270 4569 tablet_service.cc:1467] Processing CreateTablet for tablet b1f5b8ea4adc443090c7cec266f2fad4 (DEFAULT_TABLE table=test-table [id=c1ec1c3ea9be4a819f47aff93259a941]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:41:19.102003 4569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1f5b8ea4adc443090c7cec266f2fad4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:41:19.109555 4935 tablet_bootstrap.cc:492] T b1f5b8ea4adc443090c7cec266f2fad4 P c68866f8d9924715ab53ec3665024df4: Bootstrap starting.
I20260620 18:41:19.113201 4935 tablet_bootstrap.cc:654] T b1f5b8ea4adc443090c7cec266f2fad4 P c68866f8d9924715ab53ec3665024df4: Neither blocks nor log segments found. Creating new log.
I20260620 18:41:19.115075 4935 log.cc:824] T b1f5b8ea4adc443090c7cec266f2fad4 P c68866f8d9924715ab53ec3665024df4: Log is configured to *not* fsync() on all Append() calls
I20260620 18:41:19.124809 4937 tablet_bootstrap.cc:492] T b1f5b8ea4adc443090c7cec266f2fad4 P 25a89431668e4c09ab320a012c9fccc3: Bootstrap starting.
I20260620 18:41:19.129313 4935 tablet_bootstrap.cc:492] T b1f5b8ea4adc443090c7cec266f2fad4 P c68866f8d9924715ab53ec3665024df4: No bootstrap required, opened a new log
I20260620 18:41:19.129631 4935 ts_tablet_manager.cc:1397] T b1f5b8ea4adc443090c7cec266f2fad4 P c68866f8d9924715ab53ec3665024df4: Time spent bootstrapping tablet: real 0.020s user 0.008s sys 0.004s
I20260620 18:41:19.131035 4937 tablet_bootstrap.cc:654] T b1f5b8ea4adc443090c7cec266f2fad4 P 25a89431668e4c09ab320a012c9fccc3: Neither blocks nor log segments found. Creating new log.
I20260620 18:41:19.132350 4937 log.cc:824] T b1f5b8ea4adc443090c7cec266f2fad4 P 25a89431668e4c09ab320a012c9fccc3: Log is configured to *not* fsync() on all Append() calls
I20260620 18:41:19.135797 4939 tablet_bootstrap.cc:492] T b1f5b8ea4adc443090c7cec266f2fad4 P 71e5f01b9c6c460cb2557dea62904735: Bootstrap starting.
I20260620 18:41:19.139650 4939 tablet_bootstrap.cc:654] T b1f5b8ea4adc443090c7cec266f2fad4 P 71e5f01b9c6c460cb2557dea62904735: Neither blocks nor log segments found. Creating new log.
I20260620 18:41:19.140981 4939 log.cc:824] T b1f5b8ea4adc443090c7cec266f2fad4 P 71e5f01b9c6c460cb2557dea62904735: Log is configured to *not* fsync() on all Append() calls
I20260620 18:41:19.142434 4937 tablet_bootstrap.cc:492] T b1f5b8ea4adc443090c7cec266f2fad4 P 25a89431668e4c09ab320a012c9fccc3: No bootstrap required, opened a new log
I20260620 18:41:19.142772 4937 ts_tablet_manager.cc:1397] T b1f5b8ea4adc443090c7cec266f2fad4 P 25a89431668e4c09ab320a012c9fccc3: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.004s
I20260620 18:41:19.142989 4935 raft_consensus.cc:348] T b1f5b8ea4adc443090c7cec266f2fad4 P c68866f8d9924715ab53ec3665024df4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25a89431668e4c09ab320a012c9fccc3" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 46047 } } peers { permanent_uuid: "c68866f8d9924715ab53ec3665024df4" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 34019 } } peers { permanent_uuid: "71e5f01b9c6c460cb2557dea62904735" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 40059 } }
I20260620 18:41:19.143517 4935 raft_consensus.cc:374] T b1f5b8ea4adc443090c7cec266f2fad4 P c68866f8d9924715ab53ec3665024df4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:41:19.143641 4935 raft_consensus.cc:729] T b1f5b8ea4adc443090c7cec266f2fad4 P c68866f8d9924715ab53ec3665024df4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c68866f8d9924715ab53ec3665024df4, State: Initialized, Role: FOLLOWER
I20260620 18:41:19.144461 4935 consensus_queue.cc:260] T b1f5b8ea4adc443090c7cec266f2fad4 P c68866f8d9924715ab53ec3665024df4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25a89431668e4c09ab320a012c9fccc3" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 46047 } } peers { permanent_uuid: "c68866f8d9924715ab53ec3665024df4" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 34019 } } peers { permanent_uuid: "71e5f01b9c6c460cb2557dea62904735" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 40059 } }
I20260620 18:41:19.151160 4939 tablet_bootstrap.cc:492] T b1f5b8ea4adc443090c7cec266f2fad4 P 71e5f01b9c6c460cb2557dea62904735: No bootstrap required, opened a new log
I20260620 18:41:19.151574 4939 ts_tablet_manager.cc:1397] T b1f5b8ea4adc443090c7cec266f2fad4 P 71e5f01b9c6c460cb2557dea62904735: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.007s
I20260620 18:41:19.153375 4937 raft_consensus.cc:348] T b1f5b8ea4adc443090c7cec266f2fad4 P 25a89431668e4c09ab320a012c9fccc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25a89431668e4c09ab320a012c9fccc3" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 46047 } } peers { permanent_uuid: "c68866f8d9924715ab53ec3665024df4" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 34019 } } peers { permanent_uuid: "71e5f01b9c6c460cb2557dea62904735" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 40059 } }
I20260620 18:41:19.153863 4937 raft_consensus.cc:374] T b1f5b8ea4adc443090c7cec266f2fad4 P 25a89431668e4c09ab320a012c9fccc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:41:19.153995 4937 raft_consensus.cc:729] T b1f5b8ea4adc443090c7cec266f2fad4 P 25a89431668e4c09ab320a012c9fccc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25a89431668e4c09ab320a012c9fccc3, State: Initialized, Role: FOLLOWER
I20260620 18:41:19.154780 4937 consensus_queue.cc:260] T b1f5b8ea4adc443090c7cec266f2fad4 P 25a89431668e4c09ab320a012c9fccc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25a89431668e4c09ab320a012c9fccc3" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 46047 } } peers { permanent_uuid: "c68866f8d9924715ab53ec3665024df4" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 34019 } } peers { permanent_uuid: "71e5f01b9c6c460cb2557dea62904735" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 40059 } }
I20260620 18:41:19.157387 4778 heartbeater.cc:503] Master 127.23.115.62:41203 was elected leader, sending a full tablet report...
I20260620 18:41:19.164229 4935 ts_tablet_manager.cc:1428] T b1f5b8ea4adc443090c7cec266f2fad4 P c68866f8d9924715ab53ec3665024df4: Time spent starting tablet: real 0.034s user 0.011s sys 0.016s
I20260620 18:41:19.164074 4939 raft_consensus.cc:348] T b1f5b8ea4adc443090c7cec266f2fad4 P 71e5f01b9c6c460cb2557dea62904735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25a89431668e4c09ab320a012c9fccc3" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 46047 } } peers { permanent_uuid: "c68866f8d9924715ab53ec3665024df4" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 34019 } } peers { permanent_uuid: "71e5f01b9c6c460cb2557dea62904735" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 40059 } }
I20260620 18:41:19.165011 4948 raft_consensus.cc:482] T b1f5b8ea4adc443090c7cec266f2fad4 P 25a89431668e4c09ab320a012c9fccc3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:41:19.165362 4913 heartbeater.cc:503] Master 127.23.115.62:41203 was elected leader, sending a full tablet report...
I20260620 18:41:19.165303 4948 raft_consensus.cc:504] T b1f5b8ea4adc443090c7cec266f2fad4 P 25a89431668e4c09ab320a012c9fccc3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25a89431668e4c09ab320a012c9fccc3" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 46047 } } peers { permanent_uuid: "c68866f8d9924715ab53ec3665024df4" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 34019 } } peers { permanent_uuid: "71e5f01b9c6c460cb2557dea62904735" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 40059 } }
I20260620 18:41:19.166059 4937 ts_tablet_manager.cc:1428] T b1f5b8ea4adc443090c7cec266f2fad4 P 25a89431668e4c09ab320a012c9fccc3: Time spent starting tablet: real 0.023s user 0.013s sys 0.007s
I20260620 18:41:19.167820 4939 raft_consensus.cc:374] T b1f5b8ea4adc443090c7cec266f2fad4 P 71e5f01b9c6c460cb2557dea62904735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:41:19.167965 4939 raft_consensus.cc:729] T b1f5b8ea4adc443090c7cec266f2fad4 P 71e5f01b9c6c460cb2557dea62904735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e5f01b9c6c460cb2557dea62904735, State: Initialized, Role: FOLLOWER
I20260620 18:41:19.168771 4948 leader_election.cc:288] T b1f5b8ea4adc443090c7cec266f2fad4 P 25a89431668e4c09ab320a012c9fccc3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c68866f8d9924715ab53ec3665024df4 (127.23.115.2:34019), 71e5f01b9c6c460cb2557dea62904735 (127.23.115.1:40059)
I20260620 18:41:19.172572 4939 consensus_queue.cc:260] T b1f5b8ea4adc443090c7cec266f2fad4 P 71e5f01b9c6c460cb2557dea62904735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25a89431668e4c09ab320a012c9fccc3" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 46047 } } peers { permanent_uuid: "c68866f8d9924715ab53ec3665024df4" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 34019 } } peers { permanent_uuid: "71e5f01b9c6c460cb2557dea62904735" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 40059 } }
I20260620 18:41:19.175499 4917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:19.168298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.2:34019 (local address 127.23.115.3:57653)
0620 18:41:19.168493 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:19.168517 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:19.168638 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:19.169423 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:19.169428 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:19.169454 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:41:19.169659 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:19.169670 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:19.171021 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:19.171026 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:19.173000 (+ 1974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:19.173015 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:19.174721 (+ 1706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:19.174726 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:19.174916 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:19.174988 (+ 72us) client_negotiation.cc:769] Sending connection context
0620 18:41:19.175096 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 18:41:19.175234 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58,"mutex_wait_us":1}
I20260620 18:41:19.176280 4931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:19.168829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.3:57653 (local address 127.23.115.2:34019)
0620 18:41:19.169004 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:19.169010 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:19.169027 (+ 17us) server_negotiation.cc:407] Connection header received
0620 18:41:19.169100 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:19.169104 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:19.169178 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:19.169262 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:41:19.169815 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:19.170810 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:19.173210 (+ 2400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:19.174393 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:19.174546 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:19.174621 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:19.175812 (+ 1191us) server_negotiation.cc:299] Negotiation successful
0620 18:41:19.176005 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 18:41:19.177380 4933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:19.169321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.3:43329 (local address 127.23.115.1:40059)
0620 18:41:19.169503 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:19.169508 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:19.169604 (+ 96us) server_negotiation.cc:407] Connection header received
0620 18:41:19.169840 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:19.169846 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:19.169917 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:19.169976 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:41:19.170928 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:19.172192 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:19.174697 (+ 2505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:19.176123 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:19.176273 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:19.176341 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:19.176964 (+ 623us) server_negotiation.cc:299] Negotiation successful
0620 18:41:19.177116 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 18:41:19.177208 4733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1f5b8ea4adc443090c7cec266f2fad4" candidate_uuid: "25a89431668e4c09ab320a012c9fccc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c68866f8d9924715ab53ec3665024df4" is_pre_election: true
I20260620 18:41:19.177760 4733 raft_consensus.cc:2454] T b1f5b8ea4adc443090c7cec266f2fad4 P c68866f8d9924715ab53ec3665024df4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 25a89431668e4c09ab320a012c9fccc3 in term 0.
I20260620 18:41:19.178126 4950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:19.168961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.1:40059 (local address 127.23.115.3:43329)
0620 18:41:19.169499 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:19.169520 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:19.169695 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:19.170149 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:19.170153 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:19.170172 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:41:19.170525 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:19.170546 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:19.172404 (+ 1858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:19.172408 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:19.174488 (+ 2080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:19.174505 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:19.176587 (+ 2082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:19.176593 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:19.176778 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:19.176811 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 18:41:19.177555 (+ 744us) client_negotiation.cc:240] Negotiation successful
0620 18:41:19.177701 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":286,"threads_started":1}
I20260620 18:41:19.178514 4802 leader_election.cc:302] T b1f5b8ea4adc443090c7cec266f2fad4 P 25a89431668e4c09ab320a012c9fccc3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25a89431668e4c09ab320a012c9fccc3, c68866f8d9924715ab53ec3665024df4; no voters:
I20260620 18:41:19.179044 4948 raft_consensus.cc:2781] T b1f5b8ea4adc443090c7cec266f2fad4 P 25a89431668e4c09ab320a012c9fccc3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:41:19.179198 4948 raft_consensus.cc:482] T b1f5b8ea4adc443090c7cec266f2fad4 P 25a89431668e4c09ab320a012c9fccc3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:41:19.179281 4948 raft_consensus.cc:3037] T b1f5b8ea4adc443090c7cec266f2fad4 P 25a89431668e4c09ab320a012c9fccc3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:41:19.179284 4589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1f5b8ea4adc443090c7cec266f2fad4" candidate_uuid: "25a89431668e4c09ab320a012c9fccc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e5f01b9c6c460cb2557dea62904735" is_pre_election: true
I20260620 18:41:19.179831 4589 raft_consensus.cc:2454] T b1f5b8ea4adc443090c7cec266f2fad4 P 71e5f01b9c6c460cb2557dea62904735 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 25a89431668e4c09ab320a012c9fccc3 in term 0.
I20260620 18:41:19.180145 4939 ts_tablet_manager.cc:1428] T b1f5b8ea4adc443090c7cec266f2fad4 P 71e5f01b9c6c460cb2557dea62904735: Time spent starting tablet: real 0.028s user 0.013s sys 0.005s
I20260620 18:41:19.181826 4948 raft_consensus.cc:504] T b1f5b8ea4adc443090c7cec266f2fad4 P 25a89431668e4c09ab320a012c9fccc3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25a89431668e4c09ab320a012c9fccc3" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 46047 } } peers { permanent_uuid: "c68866f8d9924715ab53ec3665024df4" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 34019 } } peers { permanent_uuid: "71e5f01b9c6c460cb2557dea62904735" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 40059 } }
I20260620 18:41:19.182336 4948 leader_election.cc:288] T b1f5b8ea4adc443090c7cec266f2fad4 P 25a89431668e4c09ab320a012c9fccc3 [CANDIDATE]: Term 1 election: Requested vote from peers c68866f8d9924715ab53ec3665024df4 (127.23.115.2:34019), 71e5f01b9c6c460cb2557dea62904735 (127.23.115.1:40059)
I20260620 18:41:19.182657 4634 heartbeater.cc:503] Master 127.23.115.62:41203 was elected leader, sending a full tablet report...
I20260620 18:41:19.182808 4589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1f5b8ea4adc443090c7cec266f2fad4" candidate_uuid: "25a89431668e4c09ab320a012c9fccc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e5f01b9c6c460cb2557dea62904735"
I20260620 18:41:19.182817 4733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1f5b8ea4adc443090c7cec266f2fad4" candidate_uuid: "25a89431668e4c09ab320a012c9fccc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c68866f8d9924715ab53ec3665024df4"
I20260620 18:41:19.183068 4589 raft_consensus.cc:3037] T b1f5b8ea4adc443090c7cec266f2fad4 P 71e5f01b9c6c460cb2557dea62904735 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:41:19.183060 4733 raft_consensus.cc:3037] T b1f5b8ea4adc443090c7cec266f2fad4 P c68866f8d9924715ab53ec3665024df4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:41:19.186129 4589 raft_consensus.cc:2454] T b1f5b8ea4adc443090c7cec266f2fad4 P 71e5f01b9c6c460cb2557dea62904735 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25a89431668e4c09ab320a012c9fccc3 in term 1.
I20260620 18:41:19.186173 4733 raft_consensus.cc:2454] T b1f5b8ea4adc443090c7cec266f2fad4 P c68866f8d9924715ab53ec3665024df4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25a89431668e4c09ab320a012c9fccc3 in term 1.
I20260620 18:41:19.186575 4802 leader_election.cc:302] T b1f5b8ea4adc443090c7cec266f2fad4 P 25a89431668e4c09ab320a012c9fccc3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25a89431668e4c09ab320a012c9fccc3, c68866f8d9924715ab53ec3665024df4; no voters:
I20260620 18:41:19.186873 4948 raft_consensus.cc:2781] T b1f5b8ea4adc443090c7cec266f2fad4 P 25a89431668e4c09ab320a012c9fccc3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:41:19.187083 4948 raft_consensus.cc:686] T b1f5b8ea4adc443090c7cec266f2fad4 P 25a89431668e4c09ab320a012c9fccc3 [term 1 LEADER]: Becoming Leader. State: Replica: 25a89431668e4c09ab320a012c9fccc3, State: Running, Role: LEADER
I20260620 18:41:19.187577 4948 consensus_queue.cc:237] T b1f5b8ea4adc443090c7cec266f2fad4 P 25a89431668e4c09ab320a012c9fccc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25a89431668e4c09ab320a012c9fccc3" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 46047 } } peers { permanent_uuid: "c68866f8d9924715ab53ec3665024df4" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 34019 } } peers { permanent_uuid: "71e5f01b9c6c460cb2557dea62904735" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 40059 } }
I20260620 18:41:19.193544 4436 catalog_manager.cc:5495] T b1f5b8ea4adc443090c7cec266f2fad4 P 25a89431668e4c09ab320a012c9fccc3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 25a89431668e4c09ab320a012c9fccc3 (127.23.115.3). New cstate: current_term: 1 leader_uuid: "25a89431668e4c09ab320a012c9fccc3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25a89431668e4c09ab320a012c9fccc3" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 46047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c68866f8d9924715ab53ec3665024df4" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 34019 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71e5f01b9c6c460cb2557dea62904735" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 40059 } health_report { overall_health: UNKNOWN } } }
I20260620 18:41:19.227550 4929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:19.221966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55960 (local address 127.23.115.3:46047)
0620 18:41:19.222170 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:19.222175 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:19.222195 (+ 20us) server_negotiation.cc:407] Connection header received
0620 18:41:19.222290 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:19.222293 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:19.222368 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:19.222493 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:41:19.222849 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:19.223594 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:19.224902 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:19.225309 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:19.225441 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:19.226016 (+ 575us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:41:19.226519 (+ 503us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:41:19.226635 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:19.226894 (+ 259us) server_negotiation.cc:299] Negotiation successful
0620 18:41:19.227086 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 18:41:19.236162 4589 raft_consensus.cc:1261] T b1f5b8ea4adc443090c7cec266f2fad4 P 71e5f01b9c6c460cb2557dea62904735 [term 1 FOLLOWER]: Refusing update from remote peer 25a89431668e4c09ab320a012c9fccc3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:41:19.236264 4733 raft_consensus.cc:1261] T b1f5b8ea4adc443090c7cec266f2fad4 P c68866f8d9924715ab53ec3665024df4 [term 1 FOLLOWER]: Refusing update from remote peer 25a89431668e4c09ab320a012c9fccc3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:41:19.237102 4948 consensus_queue.cc:1035] T b1f5b8ea4adc443090c7cec266f2fad4 P 25a89431668e4c09ab320a012c9fccc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c68866f8d9924715ab53ec3665024df4" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 34019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:41:19.237406 4949 consensus_queue.cc:1035] T b1f5b8ea4adc443090c7cec266f2fad4 P 25a89431668e4c09ab320a012c9fccc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71e5f01b9c6c460cb2557dea62904735" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 40059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:41:19.250095 4954 mvcc.cc:205] Tried to move back new op lower bound from 7298993681340604416 to 7298993681157038080. Current Snapshot: MvccSnapshot[applied={T|T < 7298993681340604416}]
I20260620 18:41:19.251479 4955 mvcc.cc:205] Tried to move back new op lower bound from 7298993681340604416 to 7298993681157038080. Current Snapshot: MvccSnapshot[applied={T|T < 7298993681340604416}]
I20260620 18:41:19.268035 4435 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58708:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:41:19.268445 4435 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58708:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:41:19.273223 4435 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1be16d054bca4d19824b61372c2acb9e: Sending DeleteTablet for 3 replicas of tablet b1f5b8ea4adc443090c7cec266f2fad4
I20260620 18:41:19.274554 4847 tablet_service.cc:1514] Processing DeleteTablet for tablet b1f5b8ea4adc443090c7cec266f2fad4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:41:19 UTC) from {username='kudu', principal='kudu/127.23.115.62@KRBTEST.COM'} at 127.0.0.1:55952
I20260620 18:41:19.274751 4569 tablet_service.cc:1514] Processing DeleteTablet for tablet b1f5b8ea4adc443090c7cec266f2fad4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:41:19 UTC) from {username='kudu', principal='kudu/127.23.115.62@KRBTEST.COM'} at 127.0.0.1:58710
I20260620 18:41:19.275816 4713 tablet_service.cc:1514] Processing DeleteTablet for tablet b1f5b8ea4adc443090c7cec266f2fad4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:41:19 UTC) from {username='kudu', principal='kudu/127.23.115.62@KRBTEST.COM'} at 127.0.0.1:51596
I20260620 18:41:19.286228 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 4473
I20260620 18:41:19.287506 4964 tablet_replica.cc:330] T b1f5b8ea4adc443090c7cec266f2fad4 P 71e5f01b9c6c460cb2557dea62904735: stopping tablet replica
I20260620 18:41:19.287998 4964 raft_consensus.cc:2229] T b1f5b8ea4adc443090c7cec266f2fad4 P 71e5f01b9c6c460cb2557dea62904735 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:41:19.288560 4964 raft_consensus.cc:2258] T b1f5b8ea4adc443090c7cec266f2fad4 P 71e5f01b9c6c460cb2557dea62904735 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:41:19.291324 4964 ts_tablet_manager.cc:1905] T b1f5b8ea4adc443090c7cec266f2fad4 P 71e5f01b9c6c460cb2557dea62904735: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:41:19.306321 4933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:19.287163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58722 (local address 127.23.115.1:40059)
0620 18:41:19.287829 (+ 666us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:19.287834 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:19.287852 (+ 18us) server_negotiation.cc:407] Connection header received
0620 18:41:19.287916 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:19.287921 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:19.288002 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:19.288102 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:19.290229 (+ 2127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:19.291570 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:19.294119 (+ 2549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:19.294768 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:19.294910 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:19.298262 (+ 3352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:19.298277 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:19.298296 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:19.298335 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:19.301961 (+ 3626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:19.302844 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:19.302853 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:19.302856 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:19.303127 (+ 271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:19.303619 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:19.303624 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:19.303627 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:19.304088 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:19.304430 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:19.305699 (+ 1269us) server_negotiation.cc:299] Negotiation successful
0620 18:41:19.305923 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":550}
I20260620 18:41:19.307526 4629 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:41:19.310527 4965 tablet_replica.cc:330] T b1f5b8ea4adc443090c7cec266f2fad4 P c68866f8d9924715ab53ec3665024df4: stopping tablet replica
I20260620 18:41:19.311054 4965 raft_consensus.cc:2229] T b1f5b8ea4adc443090c7cec266f2fad4 P c68866f8d9924715ab53ec3665024df4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:41:19.311607 4965 raft_consensus.cc:2258] T b1f5b8ea4adc443090c7cec266f2fad4 P c68866f8d9924715ab53ec3665024df4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:41:19.312482 4963 tablet_replica.cc:330] T b1f5b8ea4adc443090c7cec266f2fad4 P 25a89431668e4c09ab320a012c9fccc3: stopping tablet replica
I20260620 18:41:19.312914 4963 raft_consensus.cc:2229] T b1f5b8ea4adc443090c7cec266f2fad4 P 25a89431668e4c09ab320a012c9fccc3 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:41:19.313625 4963 raft_consensus.cc:2258] T b1f5b8ea4adc443090c7cec266f2fad4 P 25a89431668e4c09ab320a012c9fccc3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:41:19.316113 4965 ts_tablet_manager.cc:1905] T b1f5b8ea4adc443090c7cec266f2fad4 P c68866f8d9924715ab53ec3665024df4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:41:19.321460 4963 ts_tablet_manager.cc:1905] T b1f5b8ea4adc443090c7cec266f2fad4 P 25a89431668e4c09ab320a012c9fccc3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:41:19.328842 4963 ts_tablet_manager.cc:1918] T b1f5b8ea4adc443090c7cec266f2fad4 P 25a89431668e4c09ab320a012c9fccc3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:41:19.329113 4963 log.cc:1196] T b1f5b8ea4adc443090c7cec266f2fad4 P 25a89431668e4c09ab320a012c9fccc3: Deleting WAL directory at /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ts-2/wal/wals/b1f5b8ea4adc443090c7cec266f2fad4
I20260620 18:41:19.329753 4963 ts_tablet_manager.cc:1939] T b1f5b8ea4adc443090c7cec266f2fad4 P 25a89431668e4c09ab320a012c9fccc3: Deleting consensus metadata
I20260620 18:41:19.331873 4423 catalog_manager.cc:4841] TS 25a89431668e4c09ab320a012c9fccc3 (127.23.115.3:46047): tablet b1f5b8ea4adc443090c7cec266f2fad4 (table test-table [id=c1ec1c3ea9be4a819f47aff93259a941]) successfully deleted
I20260620 18:41:19.335343 4965 ts_tablet_manager.cc:1918] T b1f5b8ea4adc443090c7cec266f2fad4 P c68866f8d9924715ab53ec3665024df4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:41:19.335611 4965 log.cc:1196] T b1f5b8ea4adc443090c7cec266f2fad4 P c68866f8d9924715ab53ec3665024df4: Deleting WAL directory at /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ts-1/wal/wals/b1f5b8ea4adc443090c7cec266f2fad4
I20260620 18:41:19.336156 4965 ts_tablet_manager.cc:1939] T b1f5b8ea4adc443090c7cec266f2fad4 P c68866f8d9924715ab53ec3665024df4: Deleting consensus metadata
I20260620 18:41:19.337987 4422 catalog_manager.cc:4841] TS c68866f8d9924715ab53ec3665024df4 (127.23.115.2:34019): tablet b1f5b8ea4adc443090c7cec266f2fad4 (table test-table [id=c1ec1c3ea9be4a819f47aff93259a941]) successfully deleted
I20260620 18:41:19.472582 4964 ts_tablet_manager.cc:1918] T b1f5b8ea4adc443090c7cec266f2fad4 P 71e5f01b9c6c460cb2557dea62904735: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:41:19.472816 4964 log.cc:1196] T b1f5b8ea4adc443090c7cec266f2fad4 P 71e5f01b9c6c460cb2557dea62904735: Deleting WAL directory at /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980676253068-24012-0/minicluster-data/ts-0/wal/wals/b1f5b8ea4adc443090c7cec266f2fad4
I20260620 18:41:19.473460 4964 ts_tablet_manager.cc:1939] T b1f5b8ea4adc443090c7cec266f2fad4 P 71e5f01b9c6c460cb2557dea62904735: Deleting consensus metadata
I20260620 18:41:19.475684 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 4473
I20260620 18:41:19.497368 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 4638
W20260620 18:41:19.497365 4420 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.115.1:40059 (error 108)
I20260620 18:41:19.499176 4928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:19.498611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.1:40059 (local address 127.0.0.1:58728)
0620 18:41:19.498851 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:19.499003 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.1:40059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140}
W20260620 18:41:19.500669 4420 catalog_manager.cc:4584] TS 71e5f01b9c6c460cb2557dea62904735 (127.23.115.1:40059): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b1f5b8ea4adc443090c7cec266f2fad4: Network error: Client connection negotiation failed: client connection to 127.23.115.1:40059: connect: Connection refused (error 111)
I20260620 18:41:19.513767 4931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:19.498658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51602 (local address 127.23.115.2:34019)
0620 18:41:19.498992 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:19.498998 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:19.499020 (+ 22us) server_negotiation.cc:407] Connection header received
0620 18:41:19.499086 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:19.499091 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:19.499198 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:19.499306 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:19.500608 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:19.501805 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:19.503249 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:19.503857 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:19.503984 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:19.506357 (+ 2373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:19.506372 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:19.506392 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:19.506437 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:19.510744 (+ 4307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:19.511593 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:19.511603 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:19.511606 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:19.511777 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:19.512139 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:19.512144 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:19.512148 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:19.512632 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:19.512930 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:19.513169 (+ 239us) server_negotiation.cc:299] Negotiation successful
0620 18:41:19.513396 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161}
I20260620 18:41:19.514575 4773 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:41:19.554332 4928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:19.553774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.1:40059 (local address 127.0.0.1:58744)
0620 18:41:19.554039 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:19.554148 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.1:40059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260620 18:41:19.555536 4928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:19.555141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.1:40059 (local address 127.0.0.1:58750)
0620 18:41:19.555293 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:19.555377 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.1:40059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 18:41:19.603816 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 4638
I20260620 18:41:19.616104 4928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:19.615502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.1:40059 (local address 127.0.0.1:58752)
0620 18:41:19.615699 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:19.615911 (+ 212us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.1:40059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260620 18:41:19.617097 4928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:19.616785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.1:40059 (local address 127.0.0.1:58762)
0620 18:41:19.616930 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:19.616996 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.1:40059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260620 18:41:19.624523 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 4780
I20260620 18:41:19.640543 4929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:19.625377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55966 (local address 127.23.115.3:46047)
0620 18:41:19.625551 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:19.625555 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:19.625572 (+ 17us) server_negotiation.cc:407] Connection header received
0620 18:41:19.625750 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:19.625754 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:19.625838 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:19.625921 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:19.627688 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:19.628527 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:19.630313 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:19.630777 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:19.630879 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:19.634196 (+ 3317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:19.634213 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:19.634230 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:19.634266 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:19.637428 (+ 3162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:19.638317 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:19.638324 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:19.638326 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:19.638483 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:19.638894 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:19.638897 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:19.638899 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:19.639383 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:19.639617 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:19.640032 (+ 415us) server_negotiation.cc:299] Negotiation successful
0620 18:41:19.640202 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 18:41:19.641278 4908 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:41:19.697849 4928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:19.697258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.1:40059 (local address 127.0.0.1:58764)
0620 18:41:19.697493 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:19.697631 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.1:40059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260620 18:41:19.698998 4928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:19.698527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.1:40059 (local address 127.0.0.1:58768)
0620 18:41:19.698752 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:19.698838 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.1:40059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
I20260620 18:41:19.765687 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 4780
I20260620 18:41:19.781023 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 4403
I20260620 18:41:19.782207 4466 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:41:19.891950 4928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:19.891413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.1:40059 (local address 127.0.0.1:58770)
0620 18:41:19.891636 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:19.891738 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.1:40059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 18:41:19.892222 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 4403
I20260620 18:41:19.910351 24012 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 18:41:20.246783 24012 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 18:41:20.247035 24012 mini_ranger.cc:66] Stopping Ranger...
I20260620 18:41:20.594806 24012 mini_ranger.cc:68] Stopped Ranger
2026-06-20 18:41:20.595 UTC [3455] LOG: received smart shutdown request
2026-06-20 18:41:20.598 UTC [3455] LOG: background worker "logical replication launcher" (PID 3464) exited with exit code 1
2026-06-20 18:41:20.598 UTC [3459] LOG: shutting down
2026-06-20 18:41:20.769 UTC [3455] LOG: database system is shut down
2026-06-20T18:41:20Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (55424 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980676253068-24012-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:41:20 dist-test-slave-d645 krb5kdc[4979](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:41:20 dist-test-slave-d645 krb5kdc[4979](info): set up 2 sockets
Jun 20 18:41:20 dist-test-slave-d645 krb5kdc[4979](info): commencing operation
krb5kdc: starting...
W20260620 18:41:21.936201 24012 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980676253068-24012-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980676253068-24012-0/krb5kdc/test-user.keytab.
Jun 20 18:41:22 dist-test-slave-d645 krb5kdc[4979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980882, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:41:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:41:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.115.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.115.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.115.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab.
Entry for principal kudu/127.23.115.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab.
I20260620 18:41:22.178416 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980676253068-24012-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980676253068-24012-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980676253068-24012-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.115.62:43069
--webserver_interface=127.23.115.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab
--principal=kudu/127.23.115.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:39035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.115.62:43069,127.23.115.61:46171,127.23.115.60:34687
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:41:22.551923 4992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:41:22.552248 4992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:41:22.552376 4992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:41:22.560763 4992 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:41:22.560900 4992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:41:22.560981 4992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:41:22.561055 4992 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:41:22.561112 4992 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:41:22.571419 4992 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:39035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980676253068-24012-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980676253068-24012-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.115.62:43069,127.23.115.61:46171,127.23.115.60:34687
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab
--principal=kudu/127.23.115.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.115.62:43069
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.115.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980676253068-24012-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:41:22.572748 4992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:41:22.574173 4992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:41:22.588006 4997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:22.587935 4998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:22.587970 5000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:41:22.588819 4992 server_base.cc:1034] running on GCE node
I20260620 18:41:22.589605 4992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:41:22.591063 4992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:41:22.592290 4992 hybrid_clock.cc:648] HybridClock initialized: now 1781980882592237 us; error 44 us; skew 500 ppm
I20260620 18:41:22.592669 4992 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:41:22 dist-test-slave-d645 krb5kdc[4979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980882, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:41:22.602826 4992 init.cc:377] Logged in from keytab as kudu/127.23.115.62@KRBTEST.COM (short username kudu)
I20260620 18:41:22.604704 4992 webserver.cc:460] Webserver started at http://127.23.115.62:43449/ using document root <none> and password file <none>
I20260620 18:41:22.605427 4992 fs_manager.cc:359] Metadata directory not provided
I20260620 18:41:22.605551 4992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:41:22.605952 4992 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:41:22.609139 4992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980676253068-24012-0/minicluster-data/master-0/data/instance:
uuid: "4c1e2b4439d44b6dbfa5bdc6224993f2"
format_stamp: "Formatted at 2026-06-20 18:41:22 on dist-test-slave-d645"
server_key: "1aa2a59d2da1be6b6b86fde30c22dc07"
server_key_iv: "29dd14a5be44cc8ef611bd327e4493ba"
server_key_version: "encryptionkey@0"
I20260620 18:41:22.609953 4992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980676253068-24012-0/minicluster-data/master-0/wal/instance:
uuid: "4c1e2b4439d44b6dbfa5bdc6224993f2"
format_stamp: "Formatted at 2026-06-20 18:41:22 on dist-test-slave-d645"
server_key: "1aa2a59d2da1be6b6b86fde30c22dc07"
server_key_iv: "29dd14a5be44cc8ef611bd327e4493ba"
server_key_version: "encryptionkey@0"
I20260620 18:41:22.615303 4992 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 18:41:22.618710 5007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:22.620157 4992 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:41:22.620337 4992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980676253068-24012-0/minicluster-data/master-0/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980676253068-24012-0/minicluster-data/master-0/wal
uuid: "4c1e2b4439d44b6dbfa5bdc6224993f2"
format_stamp: "Formatted at 2026-06-20 18:41:22 on dist-test-slave-d645"
server_key: "1aa2a59d2da1be6b6b86fde30c22dc07"
server_key_iv: "29dd14a5be44cc8ef611bd327e4493ba"
server_key_version: "encryptionkey@0"
I20260620 18:41:22.620522 4992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980676253068-24012-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980676253068-24012-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980676253068-24012-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:41:22.637538 4992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:41:22.642395 4992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:41:22.642690 4992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:41:22.662962 4992 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.62:43069
I20260620 18:41:22.662995 5058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.62:43069 every 8 connection(s)
I20260620 18:41:22.664625 4992 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
I20260620 18:41:22.667621 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 4992
I20260620 18:41:22.667841 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980676253068-24012-0/minicluster-data/master-0/wal/instance
I20260620 18:41:22.668747 24012 external_mini_cluster.cc:1428] Setting key 30888fb7078b944141acd7c92608f62d
I20260620 18:41:22.669062 5059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:41:22.678176 5059 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.115.62" port: 43069 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.23.115.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.115.61@KRBTEST.COM" created.
Jun 20 18:41:22 dist-test-slave-d645 krb5kdc[4979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980882, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.62@KRBTEST.COM for kudu/127.23.115.62@KRBTEST.COM
I20260620 18:41:22.702278 5062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:22.681465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:43069 (local address 127.0.0.1:54418)
0620 18:41:22.682492 (+ 1027us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:22.682544 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:22.684437 (+ 1893us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:22.685403 (+ 966us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:22.685408 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:22.686048 (+ 640us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:41:22.686463 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:22.686486 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:22.688795 (+ 2309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:22.688799 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:22.690007 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:22.690022 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:22.691310 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:22.691316 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:22.691439 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:22.692203 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:41:22.692227 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:41:22.695240 (+ 3013us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:41:22.698340 (+ 3100us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:22.698351 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:22.698364 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:22.698866 (+ 502us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:22.699314 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:22.699319 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:22.699322 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:22.699516 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:22.700307 (+ 791us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:41:22.700315 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:41:22.700553 (+ 238us) client_negotiation.cc:769] Sending connection context
0620 18:41:22.700791 (+ 238us) client_negotiation.cc:240] Negotiation successful
0620 18:41:22.701185 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":708,"thread_start_us":372,"threads_started":1}
I20260620 18:41:22.702339 5063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:22.681463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54418 (local address 127.23.115.62:43069)
0620 18:41:22.683144 (+ 1681us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:22.683152 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:22.683204 (+ 52us) server_negotiation.cc:407] Connection header received
0620 18:41:22.684821 (+ 1617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:22.684830 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:22.685138 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:22.685334 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:22.686675 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:22.688607 (+ 1932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:22.690172 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:22.691086 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:22.691286 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:22.695535 (+ 4249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:22.695545 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:22.695560 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:22.695590 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:22.698202 (+ 2612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:22.699029 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:22.699036 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:22.699038 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:22.699205 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:22.699626 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:22.699630 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:22.699633 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:22.700163 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:22.700594 (+ 431us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:22.701019 (+ 425us) server_negotiation.cc:299] Negotiation successful
0620 18:41:22.701347 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":32,"server-negotiator.queue_time_us":1502,"spinlock_wait_cycles":79232,"thread_start_us":1255,"threads_started":1}
I20260620 18:41:22.708032 5059 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.115.61" port: 46171 } has no permanent_uuid. Determining permanent_uuid...
I20260620 18:41:22.709751 5062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:22.709089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.61:46171 (local address 127.0.0.1:44720)
0620 18:41:22.709428 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:22.709580 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.61:46171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166}
W20260620 18:41:22.711275 5010 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.115.61:46171: connect: Connection refused (error 111)
I20260620 18:41:22.713188 5062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:22.712663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.61:46171 (local address 127.0.0.1:44728)
0620 18:41:22.712898 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:22.712991 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.61:46171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
W20260620 18:41:22.713511 5059 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.115.61:46171: Network error: Client connection negotiation failed: client connection to 127.23.115.61:46171: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.115.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.61.keytab.
Entry for principal kudu/127.23.115.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.61.keytab.
I20260620 18:41:22.726919 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980676253068-24012-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980676253068-24012-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980676253068-24012-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980676253068-24012-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.115.61:46171
--webserver_interface=127.23.115.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.61.keytab
--principal=kudu/127.23.115.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:39035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.115.62:43069,127.23.115.61:46171,127.23.115.60:34687
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:41:22.762900 5059 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.115.61" port: 46171 } attempt: 1
I20260620 18:41:22.764480 5062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:22.763892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.61:46171 (local address 127.0.0.1:44732)
0620 18:41:22.764167 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:22.764300 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.61:46171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260620 18:41:22.765789 5062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:22.765355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.61:46171 (local address 127.0.0.1:44746)
0620 18:41:22.765528 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:22.765622 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.61:46171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 18:41:22.765985 5059 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.115.61:46171: Network error: Client connection negotiation failed: client connection to 127.23.115.61:46171: connect: Connection refused (error 111)
I20260620 18:41:22.834278 5059 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.115.61" port: 46171 } attempt: 2
I20260620 18:41:22.835748 5062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:22.834997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.61:46171 (local address 127.0.0.1:44748)
0620 18:41:22.835437 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:22.835577 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.61:46171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260620 18:41:22.836916 5062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:22.836507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.61:46171 (local address 127.0.0.1:44750)
0620 18:41:22.836658 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:22.836758 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.61:46171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 18:41:22.837154 5059 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.115.61:46171: Network error: Client connection negotiation failed: client connection to 127.23.115.61:46171: connect: Connection refused (error 111)
I20260620 18:41:22.928452 5059 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.115.61" port: 46171 } attempt: 3
I20260620 18:41:22.930018 5062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:22.929371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.61:46171 (local address 127.0.0.1:44758)
0620 18:41:22.929668 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:22.929817 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.61:46171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260620 18:41:22.931325 5062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:22.930832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.61:46171 (local address 127.0.0.1:44768)
0620 18:41:22.931043 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:22.931154 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.61:46171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 18:41:22.931557 5059 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.115.61:46171: Network error: Client connection negotiation failed: client connection to 127.23.115.61:46171: connect: Connection refused (error 111)
W20260620 18:41:23.072911 5066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:41:23.073441 5066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:41:23.073601 5066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 18:41:23.074805 5059 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.115.61" port: 46171 } attempt: 4
I20260620 18:41:23.076087 5062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:23.075500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.61:46171 (local address 127.0.0.1:44774)
0620 18:41:23.075818 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:23.075924 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.61:46171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159}
I20260620 18:41:23.077175 5062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:23.076782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.61:46171 (local address 127.0.0.1:44788)
0620 18:41:23.076927 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:23.077032 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.61:46171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 18:41:23.077394 5059 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.115.61:46171: Network error: Client connection negotiation failed: client connection to 127.23.115.61:46171: connect: Connection refused (error 111)
W20260620 18:41:23.081812 5066 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:41:23.081916 5066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:41:23.081995 5066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:41:23.082069 5066 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:41:23.082127 5066 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:41:23.092320 5066 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:39035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980676253068-24012-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980676253068-24012-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.23.115.62:43069,127.23.115.61:46171,127.23.115.60:34687
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.61.keytab
--principal=kudu/127.23.115.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.115.61:46171
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980676253068-24012-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.23.115.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980676253068-24012-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:41:23.093641 5066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:41:23.094964 5066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:41:23.107517 5072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:23.107657 5074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:23.107503 5071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:41:23.108475 5066 server_base.cc:1034] running on GCE node
I20260620 18:41:23.109599 5066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:41:23.111080 5066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:41:23.112322 5066 hybrid_clock.cc:648] HybridClock initialized: now 1781980883112264 us; error 45 us; skew 500 ppm
I20260620 18:41:23.112709 5066 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:41:23 dist-test-slave-d645 krb5kdc[4979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980883, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:41:23.116266 5066 init.cc:377] Logged in from keytab as kudu/127.23.115.61@KRBTEST.COM (short username kudu)
I20260620 18:41:23.117864 5066 webserver.cc:460] Webserver started at http://127.23.115.61:40365/ using document root <none> and password file <none>
I20260620 18:41:23.118623 5066 fs_manager.cc:359] Metadata directory not provided
I20260620 18:41:23.118728 5066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:41:23.119047 5066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:41:23.121970 5066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980676253068-24012-0/minicluster-data/master-1/data/instance:
uuid: "10e2c8de57ac43f9a34a922fee0d65b7"
format_stamp: "Formatted at 2026-06-20 18:41:23 on dist-test-slave-d645"
server_key: "91dc4423033fc2a2eb5435e1178adc87"
server_key_iv: "69adcd8156860a4cf190d34322512b83"
server_key_version: "encryptionkey@0"
I20260620 18:41:23.122767 5066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980676253068-24012-0/minicluster-data/master-1/wal/instance:
uuid: "10e2c8de57ac43f9a34a922fee0d65b7"
format_stamp: "Formatted at 2026-06-20 18:41:23 on dist-test-slave-d645"
server_key: "91dc4423033fc2a2eb5435e1178adc87"
server_key_iv: "69adcd8156860a4cf190d34322512b83"
server_key_version: "encryptionkey@0"
I20260620 18:41:23.128191 5066 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 18:41:23.132026 5081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:23.133987 5066 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260620 18:41:23.134280 5066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980676253068-24012-0/minicluster-data/master-1/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980676253068-24012-0/minicluster-data/master-1/wal
uuid: "10e2c8de57ac43f9a34a922fee0d65b7"
format_stamp: "Formatted at 2026-06-20 18:41:23 on dist-test-slave-d645"
server_key: "91dc4423033fc2a2eb5435e1178adc87"
server_key_iv: "69adcd8156860a4cf190d34322512b83"
server_key_version: "encryptionkey@0"
I20260620 18:41:23.134500 5066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980676253068-24012-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980676253068-24012-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980676253068-24012-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:41:23.160516 5066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:41:23.166294 5066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:41:23.166555 5066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:41:23.185570 5066 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.61:46171
I20260620 18:41:23.185595 5132 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.61:46171 every 8 connection(s)
I20260620 18:41:23.186923 5066 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980676253068-24012-0/minicluster-data/master-1/data/info.pb
I20260620 18:41:23.190558 5133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:41:23.194334 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 5066
I20260620 18:41:23.194514 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980676253068-24012-0/minicluster-data/master-1/wal/instance
I20260620 18:41:23.194986 24012 external_mini_cluster.cc:1428] Setting key bbf66e092915e888c17e1fcb3da0f6ad
I20260620 18:41:23.197849 5133 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.115.62" port: 43069 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:41:23 dist-test-slave-d645 krb5kdc[4979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980883, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.61@KRBTEST.COM for kudu/127.23.115.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.115.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.115.60@KRBTEST.COM" created.
I20260620 18:41:23.220539 5063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:23.201170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54432 (local address 127.23.115.62:43069)
0620 18:41:23.201396 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:23.201402 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:23.202854 (+ 1452us) server_negotiation.cc:407] Connection header received
0620 18:41:23.204844 (+ 1990us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:23.204851 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:23.204959 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:23.205068 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:23.206705 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:23.207699 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:23.209332 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:23.209952 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:23.210070 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:23.214264 (+ 4194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:23.214270 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:23.214273 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:23.214309 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:23.217143 (+ 2834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:23.218062 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:23.218067 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:23.218070 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:23.218188 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:23.218586 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:23.218592 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:23.218594 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:23.219044 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:23.219205 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:23.219939 (+ 734us) server_negotiation.cc:299] Negotiation successful
0620 18:41:23.220216 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 18:41:23.221550 5136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:23.201789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:43069 (local address 127.0.0.1:54432)
0620 18:41:23.202671 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:23.202716 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:23.204522 (+ 1806us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:23.205341 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:23.205350 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:23.205920 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:41:23.206513 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:23.206532 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:23.207904 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:23.207909 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:23.209191 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:23.209205 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:23.210194 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:23.210199 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:23.210407 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:23.211145 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:41:23.211173 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:41:23.213977 (+ 2804us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:41:23.217303 (+ 3326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:23.217310 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:23.217329 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:23.217928 (+ 599us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:23.218294 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:23.218298 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:23.218300 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:23.218470 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:23.219182 (+ 712us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:41:23.219190 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:41:23.219557 (+ 367us) client_negotiation.cc:769] Sending connection context
0620 18:41:23.219823 (+ 266us) client_negotiation.cc:240] Negotiation successful
0620 18:41:23.220385 (+ 562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":640,"thread_start_us":415,"threads_started":1}
I20260620 18:41:23.223393 5133 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.115.61" port: 46171 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:41:23 dist-test-slave-d645 krb5kdc[4979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980883, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.61@KRBTEST.COM for kudu/127.23.115.61@KRBTEST.COM
I20260620 18:41:23.241166 5136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:23.224725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.61:46171 (local address 127.0.0.1:44798)
0620 18:41:23.225011 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:23.225029 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:23.225146 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:23.225926 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:23.225930 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:23.226351 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:41:23.226489 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:23.226498 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:23.228483 (+ 1985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:23.228490 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:23.229433 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:23.229448 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:23.230726 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:23.230732 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:23.231105 (+ 373us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:23.231689 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:41:23.231709 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:41:23.234175 (+ 2466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:41:23.238358 (+ 4183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:23.238364 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:23.238368 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:23.238909 (+ 541us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:23.239428 (+ 519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:23.239432 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:23.239435 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:23.239585 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:23.240218 (+ 633us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:41:23.240226 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:41:23.240358 (+ 132us) client_negotiation.cc:769] Sending connection context
0620 18:41:23.240475 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 18:41:23.240784 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80}
I20260620 18:41:23.241276 5138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:23.224784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44798 (local address 127.23.115.61:46171)
0620 18:41:23.225500 (+ 716us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:23.225505 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:23.225529 (+ 24us) server_negotiation.cc:407] Connection header received
0620 18:41:23.225572 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:23.225578 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:23.225777 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:23.225911 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:23.226651 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:23.228239 (+ 1588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:23.229630 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:23.230490 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:23.230664 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:23.234459 (+ 3795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:23.234479 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:23.234503 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:23.234544 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:23.238196 (+ 3652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:23.239085 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:23.239095 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:23.239098 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:23.239238 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:23.239716 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:23.239719 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:23.239721 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:23.240086 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:23.240292 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:23.240630 (+ 338us) server_negotiation.cc:299] Negotiation successful
0620 18:41:23.240901 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":581,"thread_start_us":454,"threads_started":1}
I20260620 18:41:23.244478 5133 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.115.60" port: 34687 } has no permanent_uuid. Determining permanent_uuid...
I20260620 18:41:23.245644 5136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:23.245157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.60:34687 (local address 127.0.0.1:53540)
0620 18:41:23.245352 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:23.245483 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.60:34687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.115.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.60.keytab.
Entry for principal kudu/127.23.115.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.60.keytab.
W20260620 18:41:23.245911 5085 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.115.60:34687: connect: Connection refused (error 111)
I20260620 18:41:23.247231 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980676253068-24012-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980676253068-24012-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980676253068-24012-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980676253068-24012-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.115.60:34687
--webserver_interface=127.23.115.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.60.keytab
--principal=kudu/127.23.115.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:39035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.115.62:43069,127.23.115.61:46171,127.23.115.60:34687
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:41:23.247623 5136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:23.247238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.60:34687 (local address 127.0.0.1:53548)
0620 18:41:23.247402 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:23.247496 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.60:34687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 18:41:23.247898 5133 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.115.60:34687: Network error: Client connection negotiation failed: client connection to 127.23.115.60:34687: connect: Connection refused (error 111)
I20260620 18:41:23.297262 5133 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.115.60" port: 34687 } attempt: 1
I20260620 18:41:23.298489 5136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:23.298045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.60:34687 (local address 127.0.0.1:53558)
0620 18:41:23.298227 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:23.298340 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.60:34687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260620 18:41:23.299479 5136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:23.299172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.60:34687 (local address 127.0.0.1:53564)
0620 18:41:23.299313 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:23.299381 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.60:34687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260620 18:41:23.299842 5133 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.115.60:34687: Network error: Client connection negotiation failed: client connection to 127.23.115.60:34687: connect: Connection refused (error 111)
I20260620 18:41:23.368196 5133 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.115.60" port: 34687 } attempt: 2
I20260620 18:41:23.369556 5136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:23.368910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.60:34687 (local address 127.0.0.1:53570)
0620 18:41:23.369129 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:23.369220 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.60:34687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260620 18:41:23.370920 5136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:23.370379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.60:34687 (local address 127.0.0.1:53584)
0620 18:41:23.370621 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:23.370762 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.60:34687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
W20260620 18:41:23.371363 5133 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.115.60:34687: Network error: Client connection negotiation failed: client connection to 127.23.115.60:34687: connect: Connection refused (error 111)
I20260620 18:41:23.376672 5059 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.115.61" port: 46171 } attempt: 5
Jun 20 18:41:23 dist-test-slave-d645 krb5kdc[4979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980882, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.62@KRBTEST.COM for kudu/127.23.115.61@KRBTEST.COM
I20260620 18:41:23.389510 5062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:23.377352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.61:46171 (local address 127.0.0.1:44814)
0620 18:41:23.377614 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:23.377634 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:23.377786 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:23.378111 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:23.378116 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:23.378507 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:41:23.378701 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:23.378710 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:23.379715 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:23.379721 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:23.380565 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:23.380577 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:23.381301 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:23.381307 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:23.381473 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:23.382051 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:41:23.382070 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:41:23.384306 (+ 2236us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:41:23.387039 (+ 2733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:23.387045 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:23.387049 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:23.387388 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:23.387726 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:23.387731 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:23.387734 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:23.387993 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:23.388565 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:41:23.388570 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:41:23.388714 (+ 144us) client_negotiation.cc:769] Sending connection context
0620 18:41:23.388834 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 18:41:23.389173 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99}
I20260620 18:41:23.389504 5138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:23.377467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44814 (local address 127.23.115.61:46171)
0620 18:41:23.377635 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:23.377640 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:23.377713 (+ 73us) server_negotiation.cc:407] Connection header received
0620 18:41:23.377921 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:23.377925 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:23.377986 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:23.378075 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:23.378856 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:23.379590 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:23.380700 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:23.381162 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:23.381269 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:23.384489 (+ 3220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:23.384494 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:23.384498 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:23.384529 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:23.386878 (+ 2349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:23.387498 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:23.387502 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:23.387505 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:23.387620 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:23.388142 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:23.388146 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:23.388148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:23.388453 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:23.388588 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:23.388949 (+ 361us) server_negotiation.cc:299] Negotiation successful
0620 18:41:23.389157 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 18:41:23.390695 5059 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.115.60" port: 34687 } has no permanent_uuid. Determining permanent_uuid...
I20260620 18:41:23.391759 5062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:23.391288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.60:34687 (local address 127.0.0.1:53596)
0620 18:41:23.391490 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:23.391588 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.60:34687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260620 18:41:23.392724 5062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:23.392367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.60:34687 (local address 127.0.0.1:53606)
0620 18:41:23.392513 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:23.392610 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.60:34687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 18:41:23.392949 5059 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.115.60:34687: Network error: Client connection negotiation failed: client connection to 127.23.115.60:34687: connect: Connection refused (error 111)
I20260620 18:41:23.444208 5059 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.115.60" port: 34687 } attempt: 1
I20260620 18:41:23.445844 5062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:23.445145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.60:34687 (local address 127.0.0.1:53620)
0620 18:41:23.445475 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:23.445622 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.60:34687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158}
I20260620 18:41:23.447085 5062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:23.446609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.60:34687 (local address 127.0.0.1:53624)
0620 18:41:23.446785 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:23.446901 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.60:34687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260620 18:41:23.447353 5059 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.115.60:34687: Network error: Client connection negotiation failed: client connection to 127.23.115.60:34687: connect: Connection refused (error 111)
I20260620 18:41:23.462642 5133 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.115.60" port: 34687 } attempt: 3
I20260620 18:41:23.463763 5136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:23.463303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.60:34687 (local address 127.0.0.1:53630)
0620 18:41:23.463516 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:23.463612 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.60:34687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260620 18:41:23.464841 5136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:23.464423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.60:34687 (local address 127.0.0.1:53646)
0620 18:41:23.464655 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:23.464733 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.60:34687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
W20260620 18:41:23.465060 5133 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.115.60:34687: Network error: Client connection negotiation failed: client connection to 127.23.115.60:34687: connect: Connection refused (error 111)
I20260620 18:41:23.515588 5059 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.115.60" port: 34687 } attempt: 2
I20260620 18:41:23.516871 5062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:23.516313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.60:34687 (local address 127.0.0.1:53654)
0620 18:41:23.516560 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:23.516676 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.60:34687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260620 18:41:23.517895 5062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:23.517506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.60:34687 (local address 127.0.0.1:53670)
0620 18:41:23.517676 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:23.517761 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.60:34687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 18:41:23.518131 5059 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.115.60:34687: Network error: Client connection negotiation failed: client connection to 127.23.115.60:34687: connect: Connection refused (error 111)
W20260620 18:41:23.580035 5140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:41:23.580317 5140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:41:23.580403 5140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:41:23.588327 5140 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:41:23.588445 5140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:41:23.588503 5140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:41:23.588563 5140 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:41:23.588606 5140 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:41:23.598799 5140 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:39035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980676253068-24012-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980676253068-24012-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.23.115.62:43069,127.23.115.61:46171,127.23.115.60:34687
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.60.keytab
--principal=kudu/127.23.115.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.115.60:34687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980676253068-24012-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.23.115.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980676253068-24012-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:41:23.600337 5140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:41:23.601588 5140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 18:41:23.608343 5133 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.115.60" port: 34687 } attempt: 4
I20260620 18:41:23.609433 5136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:23.608973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.60:34687 (local address 127.0.0.1:53676)
0620 18:41:23.609181 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:23.609266 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.60:34687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260620 18:41:23.610275 5136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:23.609993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.60:34687 (local address 127.0.0.1:53692)
0620 18:41:23.610117 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:23.610182 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.60:34687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260620 18:41:23.610469 5133 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.115.60:34687: Network error: Client connection negotiation failed: client connection to 127.23.115.60:34687: connect: Connection refused (error 111)
W20260620 18:41:23.613914 5148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:23.614215 5146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:41:23.614318 5140 server_base.cc:1034] running on GCE node
W20260620 18:41:23.614187 5145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:41:23.615480 5140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:41:23.616921 5140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:41:23.618149 5140 hybrid_clock.cc:648] HybridClock initialized: now 1781980883618094 us; error 45 us; skew 500 ppm
I20260620 18:41:23.618496 5140 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:41:23 dist-test-slave-d645 krb5kdc[4979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980883, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:41:23.621490 5140 init.cc:377] Logged in from keytab as kudu/127.23.115.60@KRBTEST.COM (short username kudu)
I20260620 18:41:23.622900 5140 webserver.cc:460] Webserver started at http://127.23.115.60:41839/ using document root <none> and password file <none>
I20260620 18:41:23.623723 5140 fs_manager.cc:359] Metadata directory not provided
I20260620 18:41:23.623858 5140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:41:23.624177 5140 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:41:23.624402 5059 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.115.60" port: 34687 } attempt: 3
I20260620 18:41:23.625741 5062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:23.625216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.60:34687 (local address 127.0.0.1:53708)
0620 18:41:23.625443 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:23.625545 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.60:34687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260620 18:41:23.627128 5062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:23.626597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.60:34687 (local address 127.0.0.1:53724)
0620 18:41:23.626850 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:23.626972 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.60:34687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
W20260620 18:41:23.627363 5059 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.115.60:34687: Network error: Client connection negotiation failed: client connection to 127.23.115.60:34687: connect: Connection refused (error 111)
I20260620 18:41:23.627367 5140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980676253068-24012-0/minicluster-data/master-2/data/instance:
uuid: "3b0524df396248c98713c563d8512fb4"
format_stamp: "Formatted at 2026-06-20 18:41:23 on dist-test-slave-d645"
server_key: "cc5e9861c1d6a3726d9e7725a90b87fc"
server_key_iv: "d0c0a3f3a8d4b56f7479da365a4e13ed"
server_key_version: "encryptionkey@0"
I20260620 18:41:23.628176 5140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980676253068-24012-0/minicluster-data/master-2/wal/instance:
uuid: "3b0524df396248c98713c563d8512fb4"
format_stamp: "Formatted at 2026-06-20 18:41:23 on dist-test-slave-d645"
server_key: "cc5e9861c1d6a3726d9e7725a90b87fc"
server_key_iv: "d0c0a3f3a8d4b56f7479da365a4e13ed"
server_key_version: "encryptionkey@0"
I20260620 18:41:23.633957 5140 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.005s
I20260620 18:41:23.637693 5155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:23.639364 5140 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 18:41:23.639531 5140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980676253068-24012-0/minicluster-data/master-2/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980676253068-24012-0/minicluster-data/master-2/wal
uuid: "3b0524df396248c98713c563d8512fb4"
format_stamp: "Formatted at 2026-06-20 18:41:23 on dist-test-slave-d645"
server_key: "cc5e9861c1d6a3726d9e7725a90b87fc"
server_key_iv: "d0c0a3f3a8d4b56f7479da365a4e13ed"
server_key_version: "encryptionkey@0"
I20260620 18:41:23.639709 5140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980676253068-24012-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980676253068-24012-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980676253068-24012-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:41:23.660691 5140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:41:23.665585 5140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:41:23.665886 5140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:41:23.684036 5140 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.60:34687
I20260620 18:41:23.684062 5206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.60:34687 every 8 connection(s)
I20260620 18:41:23.685408 5140 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980676253068-24012-0/minicluster-data/master-2/data/info.pb
I20260620 18:41:23.689558 5207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:41:23.689589 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 5140
I20260620 18:41:23.689766 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980676253068-24012-0/minicluster-data/master-2/wal/instance
I20260620 18:41:23.690349 24012 external_mini_cluster.cc:1428] Setting key e674b24bebfc895847b45d0f8321add6
I20260620 18:41:23.699707 5207 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.115.62" port: 43069 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:41:23 dist-test-slave-d645 krb5kdc[4979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980882, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.115.60@KRBTEST.COM
Jun 20 18:41:23 dist-test-slave-d645 krb5kdc[4979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980883, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.60@KRBTEST.COM for kudu/127.23.115.62@KRBTEST.COM
I20260620 18:41:23.715116 5210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:23.692352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53736 (local address 127.23.115.60:34687)
0620 18:41:23.693754 (+ 1402us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:23.693783 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:23.693840 (+ 57us) server_negotiation.cc:407] Connection header received
0620 18:41:23.696462 (+ 2622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:23.696486 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:23.696920 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:23.697755 (+ 835us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:23.699519 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:23.700978 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:23.702579 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:23.703562 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:23.703764 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:23.707816 (+ 4052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:23.707824 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:23.707835 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:23.707877 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:23.711138 (+ 3261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:23.711805 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:23.711812 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:23.711818 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:23.711966 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:23.712343 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:23.712347 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:23.712349 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:23.712870 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:23.713125 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:23.713592 (+ 467us) server_negotiation.cc:299] Negotiation successful
0620 18:41:23.714031 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":949,"thread_start_us":717,"threads_started":1}
I20260620 18:41:23.716190 5063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:23.702538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54446 (local address 127.23.115.62:43069)
0620 18:41:23.702747 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:23.702753 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:23.703605 (+ 852us) server_negotiation.cc:407] Connection header received
0620 18:41:23.703804 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:23.703809 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:23.703873 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:23.703954 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:23.705124 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:23.705797 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:23.707159 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:23.707625 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:23.707731 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:23.711080 (+ 3349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:23.711086 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:23.711089 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:23.711122 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:23.713458 (+ 2336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:23.714259 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:23.714263 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:23.714265 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:23.714389 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:23.714725 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:23.714728 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:23.714730 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:23.715002 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:23.715138 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:23.715648 (+ 510us) server_negotiation.cc:299] Negotiation successful
0620 18:41:23.715855 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 18:41:23.716218 5211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:23.702442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:43069 (local address 127.0.0.1:54446)
0620 18:41:23.703458 (+ 1016us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:23.703503 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:23.703670 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:23.704047 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:23.704054 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:23.704719 (+ 665us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:41:23.704977 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:23.704985 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:23.705944 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:23.705949 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:23.707039 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:23.707050 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:23.707825 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:23.707832 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:23.708009 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:23.708615 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:41:23.708637 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:41:23.710882 (+ 2245us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:41:23.713587 (+ 2705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:23.713597 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:23.713605 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:23.714149 (+ 544us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:23.714488 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:23.714493 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:23.714497 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:23.714622 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:23.715121 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:41:23.715128 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:41:23.715316 (+ 188us) client_negotiation.cc:769] Sending connection context
0620 18:41:23.715505 (+ 189us) client_negotiation.cc:240] Negotiation successful
0620 18:41:23.715807 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":751,"thread_start_us":418,"threads_started":1}
I20260620 18:41:23.717820 5207 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.115.61" port: 46171 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:41:23 dist-test-slave-d645 krb5kdc[4979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980883, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.60@KRBTEST.COM for kudu/127.23.115.61@KRBTEST.COM
Jun 20 18:41:23 dist-test-slave-d645 krb5kdc[4979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980882, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.115.62@KRBTEST.COM
I20260620 18:41:23.733070 5211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:23.718519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.61:46171 (local address 127.0.0.1:44816)
0620 18:41:23.718767 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:23.718783 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:23.718875 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:23.719182 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:23.719186 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:23.719535 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:41:23.719672 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:23.719679 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:23.720985 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:23.720989 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:23.722360 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:23.722375 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:23.723340 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:23.723346 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:23.723566 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:23.724197 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:41:23.724220 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:41:23.726777 (+ 2557us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:41:23.730371 (+ 3594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:23.730374 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:23.730377 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:23.730761 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:23.731207 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:23.731212 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:23.731216 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:23.731400 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:23.732295 (+ 895us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:41:23.732299 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:41:23.732469 (+ 170us) client_negotiation.cc:769] Sending connection context
0620 18:41:23.732583 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 18:41:23.732788 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158}
I20260620 18:41:23.733436 5138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:23.718500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44816 (local address 127.23.115.61:46171)
0620 18:41:23.718695 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:23.718701 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:23.718842 (+ 141us) server_negotiation.cc:407] Connection header received
0620 18:41:23.718992 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:23.718997 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:23.719070 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:23.719149 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:23.719806 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:23.720844 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:23.722529 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:23.723188 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:23.723323 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:23.727002 (+ 3679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:23.727010 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:23.727015 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:23.727053 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:23.730237 (+ 3184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:23.730876 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:23.730881 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:23.730885 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:23.731076 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:23.731627 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:23.731634 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:23.731637 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:23.732156 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:23.732262 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:23.732831 (+ 569us) server_negotiation.cc:299] Negotiation successful
0620 18:41:23.733110 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 18:41:23.733999 5063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:23.718122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54460 (local address 127.23.115.62:43069)
0620 18:41:23.718243 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:23.718247 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:23.718262 (+ 15us) server_negotiation.cc:407] Connection header received
0620 18:41:23.718433 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:23.718438 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:23.718506 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:23.718597 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:23.719905 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:23.720970 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:23.722661 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:23.723334 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:23.723439 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:23.727914 (+ 4475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:23.727921 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:23.727925 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:23.727963 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:23.730962 (+ 2999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:23.731816 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:23.731822 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:23.731825 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:23.731972 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:23.732432 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:23.732436 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:23.732439 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:23.732889 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:23.733173 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:23.733479 (+ 306us) server_negotiation.cc:299] Negotiation successful
0620 18:41:23.733709 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 18:41:23.734566 5207 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.115.60" port: 34687 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:41:23 dist-test-slave-d645 krb5kdc[4979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980883, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.60@KRBTEST.COM for kudu/127.23.115.60@KRBTEST.COM
Jun 20 18:41:23 dist-test-slave-d645 krb5kdc[4979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980882, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.115.61@KRBTEST.COM
I20260620 18:41:23.748175 5210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:23.735369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53744 (local address 127.23.115.60:34687)
0620 18:41:23.735801 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:23.735805 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:23.735818 (+ 13us) server_negotiation.cc:407] Connection header received
0620 18:41:23.735853 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:23.735857 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:23.735908 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:23.735990 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:23.736638 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:23.737296 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:23.738380 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:23.738776 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:23.738874 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:23.743155 (+ 4281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:23.743160 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:23.743163 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:23.743189 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:23.745747 (+ 2558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:23.746448 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:23.746452 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:23.746455 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:23.746578 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:23.746980 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:23.746986 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:23.746989 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:23.747265 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:23.747378 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:23.747766 (+ 388us) server_negotiation.cc:299] Negotiation successful
0620 18:41:23.747925 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333}
I20260620 18:41:23.748505 5211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:23.735298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.60:34687 (local address 127.0.0.1:53744)
0620 18:41:23.735460 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:23.735478 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:23.735573 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:23.736023 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:23.736027 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:23.736384 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:41:23.736518 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:23.736526 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:23.737405 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:23.737410 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:23.738262 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:23.738272 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:23.738914 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:23.738919 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:23.739117 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:23.740476 (+ 1359us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:41:23.740497 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:41:23.743000 (+ 2503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:41:23.745870 (+ 2870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:23.745873 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:23.745876 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:23.746320 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:23.746697 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:23.746701 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:23.746703 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:23.746836 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:23.747380 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:41:23.747385 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:41:23.747539 (+ 154us) client_negotiation.cc:769] Sending connection context
0620 18:41:23.747700 (+ 161us) client_negotiation.cc:240] Negotiation successful
0620 18:41:23.748114 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":86}
I20260620 18:41:23.750808 5138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:23.735646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44830 (local address 127.23.115.61:46171)
0620 18:41:23.735785 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:23.735790 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:23.735806 (+ 16us) server_negotiation.cc:407] Connection header received
0620 18:41:23.735955 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:23.735960 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:23.736022 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:23.736094 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:23.737255 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:23.738097 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:23.739609 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:23.740148 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:23.740246 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:23.745220 (+ 4974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:23.745228 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:23.745233 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:23.745272 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:23.748191 (+ 2919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:23.748941 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:23.748947 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:23.748951 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:23.749090 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:23.749482 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:23.749488 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:23.749492 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:23.749811 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:23.750022 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:23.750403 (+ 381us) server_negotiation.cc:299] Negotiation successful
0620 18:41:23.750585 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 18:41:23.754088 5207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b0524df396248c98713c563d8512fb4: Bootstrap starting.
I20260620 18:41:23.756222 24012 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:41:23.757498 5207 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b0524df396248c98713c563d8512fb4: Neither blocks nor log segments found. Creating new log.
I20260620 18:41:23.758497 5207 log.cc:824] T 00000000000000000000000000000000 P 3b0524df396248c98713c563d8512fb4: Log is configured to *not* fsync() on all Append() calls
I20260620 18:41:23.761623 5207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b0524df396248c98713c563d8512fb4: No bootstrap required, opened a new log
I20260620 18:41:23.773526 5207 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b0524df396248c98713c563d8512fb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1e2b4439d44b6dbfa5bdc6224993f2" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 43069 } } peers { permanent_uuid: "10e2c8de57ac43f9a34a922fee0d65b7" member_type: VOTER last_known_addr { host: "127.23.115.61" port: 46171 } } peers { permanent_uuid: "3b0524df396248c98713c563d8512fb4" member_type: VOTER last_known_addr { host: "127.23.115.60" port: 34687 } }
I20260620 18:41:23.775019 5207 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3b0524df396248c98713c563d8512fb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:41:23.775200 5207 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b0524df396248c98713c563d8512fb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b0524df396248c98713c563d8512fb4, State: Initialized, Role: FOLLOWER
I20260620 18:41:23.775980 5207 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b0524df396248c98713c563d8512fb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1e2b4439d44b6dbfa5bdc6224993f2" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 43069 } } peers { permanent_uuid: "10e2c8de57ac43f9a34a922fee0d65b7" member_type: VOTER last_known_addr { host: "127.23.115.61" port: 46171 } } peers { permanent_uuid: "3b0524df396248c98713c563d8512fb4" member_type: VOTER last_known_addr { host: "127.23.115.60" port: 34687 } }
I20260620 18:41:23.782235 5063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:23.760823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54474 (local address 127.23.115.62:43069)
0620 18:41:23.761003 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:23.761009 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:23.761646 (+ 637us) server_negotiation.cc:407] Connection header received
0620 18:41:23.761908 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:23.761914 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:23.762003 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:23.762091 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:23.764580 (+ 2489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:23.765765 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:23.768565 (+ 2800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:23.769337 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:23.769443 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:23.774759 (+ 5316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:23.774766 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:23.774772 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:23.774834 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:23.778523 (+ 3689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:23.779605 (+ 1082us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:23.779612 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:23.779616 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:23.779799 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:23.780277 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:23.780282 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:23.780286 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:23.780688 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:23.780933 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:23.781534 (+ 601us) server_negotiation.cc:299] Negotiation successful
0620 18:41:23.781805 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 18:41:23.783087 5220 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b0524df396248c98713c563d8512fb4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1e2b4439d44b6dbfa5bdc6224993f2" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 43069 } } peers { permanent_uuid: "10e2c8de57ac43f9a34a922fee0d65b7" member_type: VOTER last_known_addr { host: "127.23.115.61" port: 46171 } } peers { permanent_uuid: "3b0524df396248c98713c563d8512fb4" member_type: VOTER last_known_addr { host: "127.23.115.60" port: 34687 } } }
I20260620 18:41:23.783690 5220 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b0524df396248c98713c563d8512fb4 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:41:23.784885 5207 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b0524df396248c98713c563d8512fb4 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:41:23.791424 5210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:23.761651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53752 (local address 127.23.115.60:34687)
0620 18:41:23.762071 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:23.762077 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:23.775102 (+ 13025us) server_negotiation.cc:407] Connection header received
0620 18:41:23.775342 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:23.775347 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:23.775422 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:23.775485 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:23.777647 (+ 2162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:23.778830 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:23.781169 (+ 2339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:23.781833 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:23.781938 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:23.785143 (+ 3205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:23.785150 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:23.785154 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:23.785194 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:23.788349 (+ 3155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:23.789151 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:23.789157 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:23.789161 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:23.789300 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:23.789718 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:23.789724 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:23.789728 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:23.790019 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:23.790207 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:23.790713 (+ 506us) server_negotiation.cc:299] Negotiation successful
0620 18:41:23.790983 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267}
I20260620 18:41:23.794266 5138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:23.761822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44846 (local address 127.23.115.61:46171)
0620 18:41:23.762209 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:23.762214 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:23.774796 (+ 12582us) server_negotiation.cc:407] Connection header received
0620 18:41:23.775134 (+ 338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:23.775140 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:23.775228 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:23.775299 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:23.777572 (+ 2273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:23.778830 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:23.781123 (+ 2293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:23.781799 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:23.781929 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:23.787761 (+ 5832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:23.787769 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:23.787775 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:23.787824 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:23.791272 (+ 3448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:23.792088 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:23.792094 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:23.792097 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:23.792247 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:23.792745 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:23.792750 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:23.792753 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:23.793062 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:23.793295 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:23.793712 (+ 417us) server_negotiation.cc:299] Negotiation successful
0620 18:41:23.793945 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277}
W20260620 18:41:23.798758 5231 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3b0524df396248c98713c563d8512fb4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 18:41:23.799002 5231 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 18:41:23.795228 5140 master_runner.cc:186] Error getting master registration for 127.23.115.62:43069: OK, instance_id { permanent_uuid: "4c1e2b4439d44b6dbfa5bdc6224993f2" instance_seqno: 1781980882657487 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 18:41:23.804556 5059 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.115.60" port: 34687 } attempt: 4
I20260620 18:41:23.804989 5140 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Jun 20 18:41:23 dist-test-slave-d645 krb5kdc[4979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980882, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.62@KRBTEST.COM for kudu/127.23.115.60@KRBTEST.COM
I20260620 18:41:23.821288 5062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:23.805156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.60:34687 (local address 127.0.0.1:53756)
0620 18:41:23.805399 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:23.805419 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:23.805527 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:23.806102 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:23.806108 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:23.806631 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:41:23.806802 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:23.806812 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:23.807935 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:23.807941 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:23.808875 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:23.808887 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:23.809629 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:23.809634 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:23.809805 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:23.811488 (+ 1683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:41:23.811510 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:41:23.814431 (+ 2921us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:41:23.818449 (+ 4018us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:23.818453 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:23.818456 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:23.818911 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:23.819502 (+ 591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:23.819507 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:23.819510 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:23.819675 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:23.820394 (+ 719us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:41:23.820398 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:41:23.820611 (+ 213us) client_negotiation.cc:769] Sending connection context
0620 18:41:23.820750 (+ 139us) client_negotiation.cc:240] Negotiation successful
0620 18:41:23.821025 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130}
I20260620 18:41:23.822198 5210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:23.805436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53756 (local address 127.23.115.60:34687)
0620 18:41:23.805619 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:23.805625 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:23.805643 (+ 18us) server_negotiation.cc:407] Connection header received
0620 18:41:23.805700 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:23.805704 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:23.805776 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:23.805874 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:23.806973 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:23.807799 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:23.809016 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:23.809489 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:23.809600 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:23.814668 (+ 5068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:23.814674 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:23.814677 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:23.814716 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:23.818260 (+ 3544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:23.819151 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:23.819157 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:23.819160 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:23.819366 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:23.819805 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:23.819810 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:23.819813 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:23.820279 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:23.820425 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:23.821616 (+ 1191us) server_negotiation.cc:299] Negotiation successful
0620 18:41:23.821848 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 18:41:23.827207 5059 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c1e2b4439d44b6dbfa5bdc6224993f2: Bootstrap starting.
I20260620 18:41:23.832267 5059 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c1e2b4439d44b6dbfa5bdc6224993f2: Neither blocks nor log segments found. Creating new log.
I20260620 18:41:23.833307 5059 log.cc:824] T 00000000000000000000000000000000 P 4c1e2b4439d44b6dbfa5bdc6224993f2: Log is configured to *not* fsync() on all Append() calls
I20260620 18:41:23.836020 5059 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c1e2b4439d44b6dbfa5bdc6224993f2: No bootstrap required, opened a new log
I20260620 18:41:23.844313 5059 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c1e2b4439d44b6dbfa5bdc6224993f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1e2b4439d44b6dbfa5bdc6224993f2" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 43069 } } peers { permanent_uuid: "10e2c8de57ac43f9a34a922fee0d65b7" member_type: VOTER last_known_addr { host: "127.23.115.61" port: 46171 } } peers { permanent_uuid: "3b0524df396248c98713c563d8512fb4" member_type: VOTER last_known_addr { host: "127.23.115.60" port: 34687 } }
I20260620 18:41:23.844903 5059 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c1e2b4439d44b6dbfa5bdc6224993f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:41:23.845064 5059 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c1e2b4439d44b6dbfa5bdc6224993f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c1e2b4439d44b6dbfa5bdc6224993f2, State: Initialized, Role: FOLLOWER
I20260620 18:41:23.845654 5059 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c1e2b4439d44b6dbfa5bdc6224993f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1e2b4439d44b6dbfa5bdc6224993f2" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 43069 } } peers { permanent_uuid: "10e2c8de57ac43f9a34a922fee0d65b7" member_type: VOTER last_known_addr { host: "127.23.115.61" port: 46171 } } peers { permanent_uuid: "3b0524df396248c98713c563d8512fb4" member_type: VOTER last_known_addr { host: "127.23.115.60" port: 34687 } }
I20260620 18:41:23.846602 5234 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c1e2b4439d44b6dbfa5bdc6224993f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1e2b4439d44b6dbfa5bdc6224993f2" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 43069 } } peers { permanent_uuid: "10e2c8de57ac43f9a34a922fee0d65b7" member_type: VOTER last_known_addr { host: "127.23.115.61" port: 46171 } } peers { permanent_uuid: "3b0524df396248c98713c563d8512fb4" member_type: VOTER last_known_addr { host: "127.23.115.60" port: 34687 } } }
I20260620 18:41:23.847056 5234 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c1e2b4439d44b6dbfa5bdc6224993f2 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:41:23.847612 5059 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c1e2b4439d44b6dbfa5bdc6224993f2 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 18:41:23.857870 5245 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4c1e2b4439d44b6dbfa5bdc6224993f2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 18:41:23.858142 5245 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 18:41:23.857626 4992 master_runner.cc:186] Error getting master registration for 127.23.115.61:46171: OK, instance_id { permanent_uuid: "10e2c8de57ac43f9a34a922fee0d65b7" instance_seqno: 1781980883179927 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 18:41:23.871119 4992 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260620 18:41:23.895794 5013 tablet.cc:2369] T 00000000000000000000000000000000 P 4c1e2b4439d44b6dbfa5bdc6224993f2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:41:23.909708 5133 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.115.60" port: 34687 } attempt: 5
W20260620 18:41:23.918495 5161 tablet.cc:2369] T 00000000000000000000000000000000 P 3b0524df396248c98713c563d8512fb4: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 18:41:23 dist-test-slave-d645 krb5kdc[4979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980883, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.61@KRBTEST.COM for kudu/127.23.115.60@KRBTEST.COM
I20260620 18:41:23.931321 5210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:23.910988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53768 (local address 127.23.115.60:34687)
0620 18:41:23.911246 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:23.911252 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:23.911273 (+ 21us) server_negotiation.cc:407] Connection header received
0620 18:41:23.911334 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:23.911339 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:23.911422 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:23.911543 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:23.912617 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:23.913822 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:23.915697 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:23.916710 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:23.916969 (+ 259us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:23.922244 (+ 5275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:23.922255 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:23.922261 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:23.922326 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:23.927857 (+ 5531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:23.928968 (+ 1111us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:23.928974 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:23.928977 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:23.929143 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:23.929609 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:23.929614 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:23.929617 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:23.930005 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:23.930199 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:23.930610 (+ 411us) server_negotiation.cc:299] Negotiation successful
0620 18:41:23.930875 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 18:41:23.931375 5136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:23.910493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.60:34687 (local address 127.0.0.1:53768)
0620 18:41:23.910798 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:23.910820 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:23.910962 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:23.911635 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:23.911641 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:23.912211 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:41:23.912418 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:23.912431 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:23.914028 (+ 1597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:23.914034 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:23.915514 (+ 1480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:23.915531 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:23.917289 (+ 1758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:23.917298 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:23.917579 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:23.918422 (+ 843us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:41:23.918456 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:41:23.921864 (+ 3408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:41:23.928102 (+ 6238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:23.928110 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:23.928114 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:23.928805 (+ 691us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:23.929296 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:23.929301 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:23.929306 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:23.929485 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:23.930133 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:41:23.930139 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:41:23.930323 (+ 184us) client_negotiation.cc:769] Sending connection context
0620 18:41:23.930480 (+ 157us) client_negotiation.cc:240] Negotiation successful
0620 18:41:23.930993 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177}
I20260620 18:41:23.938032 5133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10e2c8de57ac43f9a34a922fee0d65b7: Bootstrap starting.
I20260620 18:41:23.942739 5133 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 10e2c8de57ac43f9a34a922fee0d65b7: Neither blocks nor log segments found. Creating new log.
I20260620 18:41:23.944176 5133 log.cc:824] T 00000000000000000000000000000000 P 10e2c8de57ac43f9a34a922fee0d65b7: Log is configured to *not* fsync() on all Append() calls
I20260620 18:41:23.947741 5133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10e2c8de57ac43f9a34a922fee0d65b7: No bootstrap required, opened a new log
I20260620 18:41:23.959051 5133 raft_consensus.cc:348] T 00000000000000000000000000000000 P 10e2c8de57ac43f9a34a922fee0d65b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1e2b4439d44b6dbfa5bdc6224993f2" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 43069 } } peers { permanent_uuid: "10e2c8de57ac43f9a34a922fee0d65b7" member_type: VOTER last_known_addr { host: "127.23.115.61" port: 46171 } } peers { permanent_uuid: "3b0524df396248c98713c563d8512fb4" member_type: VOTER last_known_addr { host: "127.23.115.60" port: 34687 } }
I20260620 18:41:23.959481 5133 raft_consensus.cc:374] T 00000000000000000000000000000000 P 10e2c8de57ac43f9a34a922fee0d65b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:41:23.959591 5133 raft_consensus.cc:729] T 00000000000000000000000000000000 P 10e2c8de57ac43f9a34a922fee0d65b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10e2c8de57ac43f9a34a922fee0d65b7, State: Initialized, Role: FOLLOWER
I20260620 18:41:23.960278 5133 consensus_queue.cc:260] T 00000000000000000000000000000000 P 10e2c8de57ac43f9a34a922fee0d65b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1e2b4439d44b6dbfa5bdc6224993f2" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 43069 } } peers { permanent_uuid: "10e2c8de57ac43f9a34a922fee0d65b7" member_type: VOTER last_known_addr { host: "127.23.115.61" port: 46171 } } peers { permanent_uuid: "3b0524df396248c98713c563d8512fb4" member_type: VOTER last_known_addr { host: "127.23.115.60" port: 34687 } }
I20260620 18:41:23.961243 5248 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10e2c8de57ac43f9a34a922fee0d65b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1e2b4439d44b6dbfa5bdc6224993f2" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 43069 } } peers { permanent_uuid: "10e2c8de57ac43f9a34a922fee0d65b7" member_type: VOTER last_known_addr { host: "127.23.115.61" port: 46171 } } peers { permanent_uuid: "3b0524df396248c98713c563d8512fb4" member_type: VOTER last_known_addr { host: "127.23.115.60" port: 34687 } } }
I20260620 18:41:23.961676 5248 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10e2c8de57ac43f9a34a922fee0d65b7 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:41:23.962796 5133 sys_catalog.cc:564] T 00000000000000000000000000000000 P 10e2c8de57ac43f9a34a922fee0d65b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:41:23.982061 5066 master_runner.cc:263] No leader master found from master 10e2c8de57ac43f9a34a922fee0d65b7
I20260620 18:41:23.982228 5066 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260620 18:41:23.987712 5259 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 10e2c8de57ac43f9a34a922fee0d65b7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 18:41:23.987962 5259 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 18:41:24.154026 5248 raft_consensus.cc:482] T 00000000000000000000000000000000 P 10e2c8de57ac43f9a34a922fee0d65b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:41:24.154381 5248 raft_consensus.cc:504] T 00000000000000000000000000000000 P 10e2c8de57ac43f9a34a922fee0d65b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1e2b4439d44b6dbfa5bdc6224993f2" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 43069 } } peers { permanent_uuid: "10e2c8de57ac43f9a34a922fee0d65b7" member_type: VOTER last_known_addr { host: "127.23.115.61" port: 46171 } } peers { permanent_uuid: "3b0524df396248c98713c563d8512fb4" member_type: VOTER last_known_addr { host: "127.23.115.60" port: 34687 } }
I20260620 18:41:24.156107 5248 leader_election.cc:288] T 00000000000000000000000000000000 P 10e2c8de57ac43f9a34a922fee0d65b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c1e2b4439d44b6dbfa5bdc6224993f2 (127.23.115.62:43069), 3b0524df396248c98713c563d8512fb4 (127.23.115.60:34687)
I20260620 18:41:24.157507 5182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "10e2c8de57ac43f9a34a922fee0d65b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b0524df396248c98713c563d8512fb4" is_pre_election: true
I20260620 18:41:24.158200 5182 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3b0524df396248c98713c563d8512fb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10e2c8de57ac43f9a34a922fee0d65b7 in term 0.
I20260620 18:41:24.159384 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "10e2c8de57ac43f9a34a922fee0d65b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c1e2b4439d44b6dbfa5bdc6224993f2" is_pre_election: true
I20260620 18:41:24.159927 5034 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4c1e2b4439d44b6dbfa5bdc6224993f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10e2c8de57ac43f9a34a922fee0d65b7 in term 0.
I20260620 18:41:24.160851 5085 leader_election.cc:302] T 00000000000000000000000000000000 P 10e2c8de57ac43f9a34a922fee0d65b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10e2c8de57ac43f9a34a922fee0d65b7, 3b0524df396248c98713c563d8512fb4; no voters:
I20260620 18:41:24.161712 5248 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 10e2c8de57ac43f9a34a922fee0d65b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:41:24.161873 5248 raft_consensus.cc:482] T 00000000000000000000000000000000 P 10e2c8de57ac43f9a34a922fee0d65b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:41:24.161972 5248 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 10e2c8de57ac43f9a34a922fee0d65b7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:41:24.165894 5248 raft_consensus.cc:504] T 00000000000000000000000000000000 P 10e2c8de57ac43f9a34a922fee0d65b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1e2b4439d44b6dbfa5bdc6224993f2" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 43069 } } peers { permanent_uuid: "10e2c8de57ac43f9a34a922fee0d65b7" member_type: VOTER last_known_addr { host: "127.23.115.61" port: 46171 } } peers { permanent_uuid: "3b0524df396248c98713c563d8512fb4" member_type: VOTER last_known_addr { host: "127.23.115.60" port: 34687 } }
I20260620 18:41:24.167273 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "10e2c8de57ac43f9a34a922fee0d65b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c1e2b4439d44b6dbfa5bdc6224993f2"
I20260620 18:41:24.167526 5034 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c1e2b4439d44b6dbfa5bdc6224993f2 [term 0 FOLLOWER]: Advancing to term 1
W20260620 18:41:24.169966 5087 tablet.cc:2369] T 00000000000000000000000000000000 P 10e2c8de57ac43f9a34a922fee0d65b7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:41:24.171188 5034 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4c1e2b4439d44b6dbfa5bdc6224993f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10e2c8de57ac43f9a34a922fee0d65b7 in term 1.
I20260620 18:41:24.171790 5085 leader_election.cc:302] T 00000000000000000000000000000000 P 10e2c8de57ac43f9a34a922fee0d65b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10e2c8de57ac43f9a34a922fee0d65b7, 4c1e2b4439d44b6dbfa5bdc6224993f2; no voters:
I20260620 18:41:24.174178 5248 leader_election.cc:288] T 00000000000000000000000000000000 P 10e2c8de57ac43f9a34a922fee0d65b7 [CANDIDATE]: Term 1 election: Requested vote from peers 4c1e2b4439d44b6dbfa5bdc6224993f2 (127.23.115.62:43069), 3b0524df396248c98713c563d8512fb4 (127.23.115.60:34687)
I20260620 18:41:24.174778 5248 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 10e2c8de57ac43f9a34a922fee0d65b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:41:24.175226 5248 raft_consensus.cc:686] T 00000000000000000000000000000000 P 10e2c8de57ac43f9a34a922fee0d65b7 [term 1 LEADER]: Becoming Leader. State: Replica: 10e2c8de57ac43f9a34a922fee0d65b7, State: Running, Role: LEADER
I20260620 18:41:24.176095 5248 consensus_queue.cc:237] T 00000000000000000000000000000000 P 10e2c8de57ac43f9a34a922fee0d65b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1e2b4439d44b6dbfa5bdc6224993f2" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 43069 } } peers { permanent_uuid: "10e2c8de57ac43f9a34a922fee0d65b7" member_type: VOTER last_known_addr { host: "127.23.115.61" port: 46171 } } peers { permanent_uuid: "3b0524df396248c98713c563d8512fb4" member_type: VOTER last_known_addr { host: "127.23.115.60" port: 34687 } }
I20260620 18:41:24.205220 5182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "10e2c8de57ac43f9a34a922fee0d65b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b0524df396248c98713c563d8512fb4"
I20260620 18:41:24.205535 5182 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b0524df396248c98713c563d8512fb4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:41:24.209894 5182 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3b0524df396248c98713c563d8512fb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10e2c8de57ac43f9a34a922fee0d65b7 in term 1.
I20260620 18:41:24.220144 5261 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10e2c8de57ac43f9a34a922fee0d65b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 10e2c8de57ac43f9a34a922fee0d65b7. Latest consensus state: current_term: 1 leader_uuid: "10e2c8de57ac43f9a34a922fee0d65b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1e2b4439d44b6dbfa5bdc6224993f2" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 43069 } } peers { permanent_uuid: "10e2c8de57ac43f9a34a922fee0d65b7" member_type: VOTER last_known_addr { host: "127.23.115.61" port: 46171 } } peers { permanent_uuid: "3b0524df396248c98713c563d8512fb4" member_type: VOTER last_known_addr { host: "127.23.115.60" port: 34687 } } }
I20260620 18:41:24.220758 5261 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10e2c8de57ac43f9a34a922fee0d65b7 [sys.catalog]: This master's current role is: LEADER
I20260620 18:41:24.228266 5263 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:41:24.230841 5263 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:41:24.255287 5182 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3b0524df396248c98713c563d8512fb4 [term 1 FOLLOWER]: Refusing update from remote peer 10e2c8de57ac43f9a34a922fee0d65b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:41:24.257514 5261 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 10e2c8de57ac43f9a34a922fee0d65b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b0524df396248c98713c563d8512fb4" member_type: VOTER last_known_addr { host: "127.23.115.60" port: 34687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:41:24.267159 5034 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4c1e2b4439d44b6dbfa5bdc6224993f2 [term 1 FOLLOWER]: Refusing update from remote peer 10e2c8de57ac43f9a34a922fee0d65b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:41:24.268306 5261 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 10e2c8de57ac43f9a34a922fee0d65b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c1e2b4439d44b6dbfa5bdc6224993f2" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 43069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:41:24.297832 5234 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c1e2b4439d44b6dbfa5bdc6224993f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 10e2c8de57ac43f9a34a922fee0d65b7. Latest consensus state: current_term: 1 leader_uuid: "10e2c8de57ac43f9a34a922fee0d65b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1e2b4439d44b6dbfa5bdc6224993f2" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 43069 } } peers { permanent_uuid: "10e2c8de57ac43f9a34a922fee0d65b7" member_type: VOTER last_known_addr { host: "127.23.115.61" port: 46171 } } peers { permanent_uuid: "3b0524df396248c98713c563d8512fb4" member_type: VOTER last_known_addr { host: "127.23.115.60" port: 34687 } } }
I20260620 18:41:24.298202 5234 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c1e2b4439d44b6dbfa5bdc6224993f2 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:41:24.309274 5264 mvcc.cc:205] Tried to move back new op lower bound from 7298993701891915776 to 7298993701619249152. Current Snapshot: MvccSnapshot[applied={T|T < 7298993701891915776}]
I20260620 18:41:24.315541 5261 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10e2c8de57ac43f9a34a922fee0d65b7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "10e2c8de57ac43f9a34a922fee0d65b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1e2b4439d44b6dbfa5bdc6224993f2" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 43069 } } peers { permanent_uuid: "10e2c8de57ac43f9a34a922fee0d65b7" member_type: VOTER last_known_addr { host: "127.23.115.61" port: 46171 } } peers { permanent_uuid: "3b0524df396248c98713c563d8512fb4" member_type: VOTER last_known_addr { host: "127.23.115.60" port: 34687 } } }
I20260620 18:41:24.315899 5261 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10e2c8de57ac43f9a34a922fee0d65b7 [sys.catalog]: This master's current role is: LEADER
I20260620 18:41:24.316855 5266 mvcc.cc:205] Tried to move back new op lower bound from 7298993701891915776 to 7298993701619249152. Current Snapshot: MvccSnapshot[applied={T|T < 7298993701891915776}]
I20260620 18:41:24.318218 5263 catalog_manager.cc:1347] Generated new cluster ID: a26f9a65e3fc418ba9b4b9d3b9439210
I20260620 18:41:24.318343 5263 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:41:24.320255 5220 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b0524df396248c98713c563d8512fb4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 10e2c8de57ac43f9a34a922fee0d65b7. Latest consensus state: current_term: 1 leader_uuid: "10e2c8de57ac43f9a34a922fee0d65b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1e2b4439d44b6dbfa5bdc6224993f2" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 43069 } } peers { permanent_uuid: "10e2c8de57ac43f9a34a922fee0d65b7" member_type: VOTER last_known_addr { host: "127.23.115.61" port: 46171 } } peers { permanent_uuid: "3b0524df396248c98713c563d8512fb4" member_type: VOTER last_known_addr { host: "127.23.115.60" port: 34687 } } }
I20260620 18:41:24.320566 5220 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b0524df396248c98713c563d8512fb4 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:41:24.323705 5234 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c1e2b4439d44b6dbfa5bdc6224993f2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "10e2c8de57ac43f9a34a922fee0d65b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1e2b4439d44b6dbfa5bdc6224993f2" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 43069 } } peers { permanent_uuid: "10e2c8de57ac43f9a34a922fee0d65b7" member_type: VOTER last_known_addr { host: "127.23.115.61" port: 46171 } } peers { permanent_uuid: "3b0524df396248c98713c563d8512fb4" member_type: VOTER last_known_addr { host: "127.23.115.60" port: 34687 } } }
I20260620 18:41:24.323980 5234 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c1e2b4439d44b6dbfa5bdc6224993f2 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:41:24.324291 5248 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10e2c8de57ac43f9a34a922fee0d65b7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "10e2c8de57ac43f9a34a922fee0d65b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1e2b4439d44b6dbfa5bdc6224993f2" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 43069 } } peers { permanent_uuid: "10e2c8de57ac43f9a34a922fee0d65b7" member_type: VOTER last_known_addr { host: "127.23.115.61" port: 46171 } } peers { permanent_uuid: "3b0524df396248c98713c563d8512fb4" member_type: VOTER last_known_addr { host: "127.23.115.60" port: 34687 } } }
I20260620 18:41:24.324641 5248 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10e2c8de57ac43f9a34a922fee0d65b7 [sys.catalog]: This master's current role is: LEADER
I20260620 18:41:24.326455 5265 mvcc.cc:205] Tried to move back new op lower bound from 7298993701891915776 to 7298993701619249152. Current Snapshot: MvccSnapshot[applied={T|T < 7298993701891915776}]
I20260620 18:41:24.327134 5220 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b0524df396248c98713c563d8512fb4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "10e2c8de57ac43f9a34a922fee0d65b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1e2b4439d44b6dbfa5bdc6224993f2" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 43069 } } peers { permanent_uuid: "10e2c8de57ac43f9a34a922fee0d65b7" member_type: VOTER last_known_addr { host: "127.23.115.61" port: 46171 } } peers { permanent_uuid: "3b0524df396248c98713c563d8512fb4" member_type: VOTER last_known_addr { host: "127.23.115.60" port: 34687 } } }
I20260620 18:41:24.327349 5220 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b0524df396248c98713c563d8512fb4 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:41:24.344880 5263 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:41:24.346611 5263 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:41:24.362758 5263 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 10e2c8de57ac43f9a34a922fee0d65b7: Generated new TSK 0
I20260620 18:41:24.363878 5263 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:41:24.800954 5231 catalog_manager.cc:1259] Loaded cluster ID: a26f9a65e3fc418ba9b4b9d3b9439210
I20260620 18:41:24.801060 5231 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3b0524df396248c98713c563d8512fb4: loading cluster ID for follower catalog manager: success
I20260620 18:41:24.803930 5231 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3b0524df396248c98713c563d8512fb4: acquiring CA information for follower catalog manager: success
I20260620 18:41:24.805738 5231 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3b0524df396248c98713c563d8512fb4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 18:41:24.860284 5245 catalog_manager.cc:1259] Loaded cluster ID: a26f9a65e3fc418ba9b4b9d3b9439210
I20260620 18:41:24.860388 5245 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4c1e2b4439d44b6dbfa5bdc6224993f2: loading cluster ID for follower catalog manager: success
I20260620 18:41:24.863127 5245 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4c1e2b4439d44b6dbfa5bdc6224993f2: acquiring CA information for follower catalog manager: success
I20260620 18:41:24.864981 5245 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4c1e2b4439d44b6dbfa5bdc6224993f2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 18:41:24.941712 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 4992
I20260620 18:41:24.943672 5054 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:41:25.057041 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 4992
I20260620 18:41:25.074992 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 5066
I20260620 18:41:25.076141 5128 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:41:25.183564 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 5066
I20260620 18:41:25.200929 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 5140
I20260620 18:41:25.202004 5202 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:41:25.322844 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 5140
2026-06-20T18:41:25Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4431 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980676253068-24012-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:41:25 dist-test-slave-d645 krb5kdc[5281](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:41:25 dist-test-slave-d645 krb5kdc[5281](info): set up 2 sockets
Jun 20 18:41:25 dist-test-slave-d645 krb5kdc[5281](info): commencing operation
krb5kdc: starting...
W20260620 18:41:26.367486 24012 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.008s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980676253068-24012-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980676253068-24012-0/krb5kdc/test-user.keytab.
Jun 20 18:41:26 dist-test-slave-d645 krb5kdc[5281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:41:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:41:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.115.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.115.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.115.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab.
Entry for principal kudu/127.23.115.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab.
I20260620 18:41:26.599390 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980676253068-24012-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980676253068-24012-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980676253068-24012-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.115.62:36645
--webserver_interface=127.23.115.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab
--principal=kudu/127.23.115.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:42193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.115.62:36645
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:41:26.934576 5294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:41:26.934886 5294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:41:26.935027 5294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:41:26.942401 5294 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:41:26.942519 5294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:41:26.942577 5294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:41:26.942657 5294 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:41:26.942714 5294 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:41:26.955533 5294 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:42193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980676253068-24012-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980676253068-24012-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.115.62:36645
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab
--principal=kudu/127.23.115.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.115.62:36645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.115.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980676253068-24012-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:41:26.956902 5294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:41:26.958181 5294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:41:26.970779 5299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:26.970762 5300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:26.971052 5302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:41:26.971750 5294 server_base.cc:1034] running on GCE node
I20260620 18:41:26.972524 5294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:41:26.973969 5294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:41:26.975262 5294 hybrid_clock.cc:648] HybridClock initialized: now 1781980886975150 us; error 84 us; skew 500 ppm
I20260620 18:41:26.975821 5294 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:41:26 dist-test-slave-d645 krb5kdc[5281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980886, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:41:26.979844 5294 init.cc:377] Logged in from keytab as kudu/127.23.115.62@KRBTEST.COM (short username kudu)
I20260620 18:41:26.981593 5294 webserver.cc:460] Webserver started at http://127.23.115.62:43031/ using document root <none> and password file <none>
I20260620 18:41:26.982300 5294 fs_manager.cc:359] Metadata directory not provided
I20260620 18:41:26.982422 5294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:41:26.982722 5294 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:41:26.985970 5294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980676253068-24012-0/minicluster-data/master-0/data/instance:
uuid: "ca706ace1be24b2ba8292b11526a3d2e"
format_stamp: "Formatted at 2026-06-20 18:41:26 on dist-test-slave-d645"
server_key: "67d8b752747ab582d29e982a605bae50"
server_key_iv: "77d001d26af057128e27aa286e7db6a6"
server_key_version: "encryptionkey@0"
I20260620 18:41:26.986704 5294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980676253068-24012-0/minicluster-data/master-0/wal/instance:
uuid: "ca706ace1be24b2ba8292b11526a3d2e"
format_stamp: "Formatted at 2026-06-20 18:41:26 on dist-test-slave-d645"
server_key: "67d8b752747ab582d29e982a605bae50"
server_key_iv: "77d001d26af057128e27aa286e7db6a6"
server_key_version: "encryptionkey@0"
I20260620 18:41:26.991974 5294 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260620 18:41:26.996258 5309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:26.997838 5294 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 18:41:26.998018 5294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980676253068-24012-0/minicluster-data/master-0/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980676253068-24012-0/minicluster-data/master-0/wal
uuid: "ca706ace1be24b2ba8292b11526a3d2e"
format_stamp: "Formatted at 2026-06-20 18:41:26 on dist-test-slave-d645"
server_key: "67d8b752747ab582d29e982a605bae50"
server_key_iv: "77d001d26af057128e27aa286e7db6a6"
server_key_version: "encryptionkey@0"
I20260620 18:41:26.998247 5294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980676253068-24012-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980676253068-24012-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980676253068-24012-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:41:27.018075 5294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:41:27.024153 5294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:41:27.024443 5294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:41:27.043977 5294 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.62:36645
I20260620 18:41:27.044108 5360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.62:36645 every 8 connection(s)
I20260620 18:41:27.045389 5294 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
I20260620 18:41:27.049311 5361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:41:27.053645 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 5294
I20260620 18:41:27.053886 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980676253068-24012-0/minicluster-data/master-0/wal/instance
I20260620 18:41:27.054668 24012 external_mini_cluster.cc:1428] Setting key 4df29d785e509fa8f8b4b2004a71847a
I20260620 18:41:27.061157 5361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca706ace1be24b2ba8292b11526a3d2e: Bootstrap starting.
I20260620 18:41:27.065922 5361 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca706ace1be24b2ba8292b11526a3d2e: Neither blocks nor log segments found. Creating new log.
I20260620 18:41:27.067271 5361 log.cc:824] T 00000000000000000000000000000000 P ca706ace1be24b2ba8292b11526a3d2e: Log is configured to *not* fsync() on all Append() calls
I20260620 18:41:27.070691 5361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca706ace1be24b2ba8292b11526a3d2e: No bootstrap required, opened a new log
Jun 20 18:41:27 dist-test-slave-d645 krb5kdc[5281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.115.62@KRBTEST.COM
I20260620 18:41:27.078847 5361 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca706ace1be24b2ba8292b11526a3d2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca706ace1be24b2ba8292b11526a3d2e" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 36645 } }
I20260620 18:41:27.079293 5361 raft_consensus.cc:374] T 00000000000000000000000000000000 P ca706ace1be24b2ba8292b11526a3d2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:41:27.079412 5361 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca706ace1be24b2ba8292b11526a3d2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca706ace1be24b2ba8292b11526a3d2e, State: Initialized, Role: FOLLOWER
I20260620 18:41:27.080047 5361 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca706ace1be24b2ba8292b11526a3d2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca706ace1be24b2ba8292b11526a3d2e" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 36645 } }
I20260620 18:41:27.080310 5361 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca706ace1be24b2ba8292b11526a3d2e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:41:27.080446 5361 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca706ace1be24b2ba8292b11526a3d2e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:41:27.080612 5361 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca706ace1be24b2ba8292b11526a3d2e [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:41:27.080771 5364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:27.056712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49008 (local address 127.23.115.62:36645)
0620 18:41:27.057494 (+ 782us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:27.057505 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:27.057548 (+ 43us) server_negotiation.cc:407] Connection header received
0620 18:41:27.059895 (+ 2347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:27.059921 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:27.060429 (+ 508us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:27.060815 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:27.062633 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:27.064079 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:27.066470 (+ 2391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:27.067302 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:27.067483 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:27.072509 (+ 5026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:27.072521 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:27.072537 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:27.072597 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:27.076317 (+ 3720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:27.077084 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:27.077092 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:27.077099 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:27.077276 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:27.077657 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:27.077661 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:27.077663 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:27.078351 (+ 688us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:27.078639 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:27.079184 (+ 545us) server_negotiation.cc:299] Negotiation successful
0620 18:41:27.079641 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":363,"threads_started":1}
I20260620 18:41:27.083659 5361 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca706ace1be24b2ba8292b11526a3d2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca706ace1be24b2ba8292b11526a3d2e" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 36645 } }
I20260620 18:41:27.084187 5361 leader_election.cc:302] T 00000000000000000000000000000000 P ca706ace1be24b2ba8292b11526a3d2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca706ace1be24b2ba8292b11526a3d2e; no voters:
I20260620 18:41:27.084807 5361 leader_election.cc:288] T 00000000000000000000000000000000 P ca706ace1be24b2ba8292b11526a3d2e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:41:27.084942 5366 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca706ace1be24b2ba8292b11526a3d2e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:41:27.085229 24012 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:41:27.087204 5366 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca706ace1be24b2ba8292b11526a3d2e [term 1 LEADER]: Becoming Leader. State: Replica: ca706ace1be24b2ba8292b11526a3d2e, State: Running, Role: LEADER
I20260620 18:41:27.087891 5366 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca706ace1be24b2ba8292b11526a3d2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca706ace1be24b2ba8292b11526a3d2e" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 36645 } }
I20260620 18:41:27.090128 5366 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca706ace1be24b2ba8292b11526a3d2e [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca706ace1be24b2ba8292b11526a3d2e. Latest consensus state: current_term: 1 leader_uuid: "ca706ace1be24b2ba8292b11526a3d2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca706ace1be24b2ba8292b11526a3d2e" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 36645 } } }
I20260620 18:41:27.090477 5366 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca706ace1be24b2ba8292b11526a3d2e [sys.catalog]: This master's current role is: LEADER
I20260620 18:41:27.090835 5368 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca706ace1be24b2ba8292b11526a3d2e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ca706ace1be24b2ba8292b11526a3d2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca706ace1be24b2ba8292b11526a3d2e" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 36645 } } }
I20260620 18:41:27.091194 5368 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca706ace1be24b2ba8292b11526a3d2e [sys.catalog]: This master's current role is: LEADER
I20260620 18:41:27.092041 5361 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca706ace1be24b2ba8292b11526a3d2e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:41:27.092223 5370 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:41:27.097951 5370 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:41:27.108577 5370 catalog_manager.cc:1347] Generated new cluster ID: 00cff9b5427a43148c6d5ec0fb61cfc0
I20260620 18:41:27.108702 5370 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:41:27.124442 5370 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:41:27.125758 5370 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:41:27.133276 5370 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ca706ace1be24b2ba8292b11526a3d2e: Generated new TSK 0
I20260620 18:41:27.134097 5370 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:41:27.147094 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 5294
I20260620 18:41:27.148768 5356 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:41:27.242743 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 5294
2026-06-20T18:41:27Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1916 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:41:27 dist-test-slave-d645 krb5kdc[5389](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:41:27 dist-test-slave-d645 krb5kdc[5389](info): set up 2 sockets
Jun 20 18:41:27 dist-test-slave-d645 krb5kdc[5389](info): commencing operation
krb5kdc: starting...
W20260620 18:41:28.291450 24012 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/krb5kdc/test-user.keytab.
Jun 20 18:41:28 dist-test-slave-d645 krb5kdc[5389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980888, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:41:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:41:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.115.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.115.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.115.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab.
Entry for principal kudu/127.23.115.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab.
I20260620 18:41:28.522542 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.115.62:35467
--webserver_interface=127.23.115.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab
--principal=kudu/127.23.115.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:33195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.115.62:35467
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:41:28.854919 5402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:41:28.855247 5402 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:41:28.855343 5402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:41:28.862658 5402 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:41:28.862774 5402 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:41:28.862826 5402 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:41:28.862882 5402 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:41:28.862947 5402 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:41:28.872941 5402 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:33195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.115.62:35467
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab
--principal=kudu/127.23.115.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.115.62:35467
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.115.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:41:28.874207 5402 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:41:28.875694 5402 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:41:28.887990 5410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:28.888019 5408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:28.888057 5407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:41:28.888928 5402 server_base.cc:1034] running on GCE node
I20260620 18:41:28.889734 5402 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:41:28.891218 5402 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:41:28.892428 5402 hybrid_clock.cc:648] HybridClock initialized: now 1781980888892382 us; error 36 us; skew 500 ppm
I20260620 18:41:28.892783 5402 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:41:28 dist-test-slave-d645 krb5kdc[5389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980888, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:41:28.900970 5402 init.cc:377] Logged in from keytab as kudu/127.23.115.62@KRBTEST.COM (short username kudu)
I20260620 18:41:28.902602 5402 webserver.cc:460] Webserver started at http://127.23.115.62:45795/ using document root <none> and password file <none>
I20260620 18:41:28.903383 5402 fs_manager.cc:359] Metadata directory not provided
I20260620 18:41:28.903496 5402 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:41:28.903784 5402 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:41:28.906698 5402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/master-0/data/instance:
uuid: "a15618092ac84abc82a0f318082d7455"
format_stamp: "Formatted at 2026-06-20 18:41:28 on dist-test-slave-d645"
server_key: "a4bfc05ff5542979236ba117c8b09d67"
server_key_iv: "18565b14a5c71413dce244e0ab893eac"
server_key_version: "encryptionkey@0"
I20260620 18:41:28.907472 5402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/master-0/wal/instance:
uuid: "a15618092ac84abc82a0f318082d7455"
format_stamp: "Formatted at 2026-06-20 18:41:28 on dist-test-slave-d645"
server_key: "a4bfc05ff5542979236ba117c8b09d67"
server_key_iv: "18565b14a5c71413dce244e0ab893eac"
server_key_version: "encryptionkey@0"
I20260620 18:41:28.912910 5402 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:41:28.916430 5417 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:28.918273 5402 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 18:41:28.918449 5402 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/master-0/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/master-0/wal
uuid: "a15618092ac84abc82a0f318082d7455"
format_stamp: "Formatted at 2026-06-20 18:41:28 on dist-test-slave-d645"
server_key: "a4bfc05ff5542979236ba117c8b09d67"
server_key_iv: "18565b14a5c71413dce244e0ab893eac"
server_key_version: "encryptionkey@0"
I20260620 18:41:28.918643 5402 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:41:28.948359 5402 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:41:28.952505 5402 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:41:28.952776 5402 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:41:28.970273 5402 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.62:35467
I20260620 18:41:28.970322 5468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.62:35467 every 8 connection(s)
I20260620 18:41:28.971889 5402 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
I20260620 18:41:28.974901 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 5402
I20260620 18:41:28.975157 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/master-0/wal/instance
I20260620 18:41:28.975924 24012 external_mini_cluster.cc:1428] Setting key 8e95ea75df7e035309418b3de29ab74d
I20260620 18:41:28.975967 5469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:41:28.991012 5469 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a15618092ac84abc82a0f318082d7455: Bootstrap starting.
Jun 20 18:41:28 dist-test-slave-d645 krb5kdc[5389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980888, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.115.62@KRBTEST.COM
I20260620 18:41:28.994426 5469 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a15618092ac84abc82a0f318082d7455: Neither blocks nor log segments found. Creating new log.
I20260620 18:41:28.995558 5469 log.cc:824] T 00000000000000000000000000000000 P a15618092ac84abc82a0f318082d7455: Log is configured to *not* fsync() on all Append() calls
I20260620 18:41:28.998337 5469 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a15618092ac84abc82a0f318082d7455: No bootstrap required, opened a new log
I20260620 18:41:29.000152 5472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:28.977779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41320 (local address 127.23.115.62:35467)
0620 18:41:28.978835 (+ 1056us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:28.978847 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:28.978900 (+ 53us) server_negotiation.cc:407] Connection header received
0620 18:41:28.980779 (+ 1879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:28.980800 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:28.981205 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:28.981525 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:28.983672 (+ 2147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:28.985203 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:28.987121 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:28.987900 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:28.988058 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:28.992915 (+ 4857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:28.992925 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:28.992941 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:28.992991 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:28.996119 (+ 3128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:28.996853 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:28.996860 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:28.996866 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:28.997041 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:28.997420 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:28.997423 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:28.997426 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:28.997954 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:28.998261 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:28.998698 (+ 437us) server_negotiation.cc:299] Negotiation successful
0620 18:41:28.999181 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":856,"thread_start_us":468,"threads_started":1}
I20260620 18:41:29.006261 5469 raft_consensus.cc:348] T 00000000000000000000000000000000 P a15618092ac84abc82a0f318082d7455 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a15618092ac84abc82a0f318082d7455" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 35467 } }
I20260620 18:41:29.006655 5469 raft_consensus.cc:374] T 00000000000000000000000000000000 P a15618092ac84abc82a0f318082d7455 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:41:29.006759 5469 raft_consensus.cc:729] T 00000000000000000000000000000000 P a15618092ac84abc82a0f318082d7455 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a15618092ac84abc82a0f318082d7455, State: Initialized, Role: FOLLOWER
I20260620 18:41:29.007918 5469 consensus_queue.cc:260] T 00000000000000000000000000000000 P a15618092ac84abc82a0f318082d7455 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a15618092ac84abc82a0f318082d7455" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 35467 } }
I20260620 18:41:29.008245 5469 raft_consensus.cc:388] T 00000000000000000000000000000000 P a15618092ac84abc82a0f318082d7455 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:41:29.008411 5469 raft_consensus.cc:482] T 00000000000000000000000000000000 P a15618092ac84abc82a0f318082d7455 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:41:29.008605 5469 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a15618092ac84abc82a0f318082d7455 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:41:29.010974 5469 raft_consensus.cc:504] T 00000000000000000000000000000000 P a15618092ac84abc82a0f318082d7455 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a15618092ac84abc82a0f318082d7455" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 35467 } }
I20260620 18:41:29.011507 5469 leader_election.cc:302] T 00000000000000000000000000000000 P a15618092ac84abc82a0f318082d7455 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a15618092ac84abc82a0f318082d7455; no voters:
I20260620 18:41:29.012228 5469 leader_election.cc:288] T 00000000000000000000000000000000 P a15618092ac84abc82a0f318082d7455 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:41:29.012503 5474 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a15618092ac84abc82a0f318082d7455 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:41:29.013365 5474 raft_consensus.cc:686] T 00000000000000000000000000000000 P a15618092ac84abc82a0f318082d7455 [term 1 LEADER]: Becoming Leader. State: Replica: a15618092ac84abc82a0f318082d7455, State: Running, Role: LEADER
I20260620 18:41:29.013916 5474 consensus_queue.cc:237] T 00000000000000000000000000000000 P a15618092ac84abc82a0f318082d7455 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a15618092ac84abc82a0f318082d7455" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 35467 } }
I20260620 18:41:29.014637 5469 sys_catalog.cc:564] T 00000000000000000000000000000000 P a15618092ac84abc82a0f318082d7455 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:41:29.017218 5475 sys_catalog.cc:455] T 00000000000000000000000000000000 P a15618092ac84abc82a0f318082d7455 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a15618092ac84abc82a0f318082d7455" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a15618092ac84abc82a0f318082d7455" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 35467 } } }
I20260620 18:41:29.017549 5475 sys_catalog.cc:458] T 00000000000000000000000000000000 P a15618092ac84abc82a0f318082d7455 [sys.catalog]: This master's current role is: LEADER
I20260620 18:41:29.017760 5476 sys_catalog.cc:455] T 00000000000000000000000000000000 P a15618092ac84abc82a0f318082d7455 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a15618092ac84abc82a0f318082d7455. Latest consensus state: current_term: 1 leader_uuid: "a15618092ac84abc82a0f318082d7455" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a15618092ac84abc82a0f318082d7455" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 35467 } } }
I20260620 18:41:29.018226 5476 sys_catalog.cc:458] T 00000000000000000000000000000000 P a15618092ac84abc82a0f318082d7455 [sys.catalog]: This master's current role is: LEADER
I20260620 18:41:29.021198 5479 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:41:29.026198 5479 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:41:29.035558 5479 catalog_manager.cc:1347] Generated new cluster ID: 2ce60297219545ebaf2008b08b4ce1ed
I20260620 18:41:29.035681 5479 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:41:29.045867 5479 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:41:29.047793 5479 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:41:29.055665 5479 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a15618092ac84abc82a0f318082d7455: Generated new TSK 0
I20260620 18:41:29.056787 5479 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.115.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.115.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.115.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.1.keytab.
Entry for principal kudu/127.23.115.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.1.keytab.
I20260620 18:41:29.111280 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.115.1:0
--local_ip_for_outbound_sockets=127.23.115.1
--webserver_interface=127.23.115.1
--webserver_port=0
--tserver_master_addrs=127.23.115.62:35467
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.1.keytab
--principal=kudu/127.23.115.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:33195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:41:29.439188 5495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:41:29.439457 5495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:41:29.439550 5495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:41:29.446632 5495 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:41:29.446748 5495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:41:29.446859 5495 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.115.1
I20260620 18:41:29.456615 5495 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:33195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.1.keytab
--principal=kudu/127.23.115.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.115.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.115.1
--webserver_port=0
--tserver_master_addrs=127.23.115.62:35467
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.115.1
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:41:29.457824 5495 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:41:29.459072 5495 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:41:29.471881 5503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:29.471916 5501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:29.472157 5500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:41:29.472641 5495 server_base.cc:1034] running on GCE node
I20260620 18:41:29.473212 5495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:41:29.474165 5495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:41:29.475364 5495 hybrid_clock.cc:648] HybridClock initialized: now 1781980889475318 us; error 39 us; skew 500 ppm
I20260620 18:41:29.475668 5495 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:41:29 dist-test-slave-d645 krb5kdc[5389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980889, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:41:29.478888 5495 init.cc:377] Logged in from keytab as kudu/127.23.115.1@KRBTEST.COM (short username kudu)
I20260620 18:41:29.480384 5495 webserver.cc:460] Webserver started at http://127.23.115.1:43901/ using document root <none> and password file <none>
I20260620 18:41:29.481034 5495 fs_manager.cc:359] Metadata directory not provided
I20260620 18:41:29.481161 5495 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:41:29.481437 5495 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:41:29.484367 5495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-0/data/instance:
uuid: "f758242b8e3d4b2e8742c80587dbc12c"
format_stamp: "Formatted at 2026-06-20 18:41:29 on dist-test-slave-d645"
server_key: "e7283e95ed8c97ee2dd7a36dcede1350"
server_key_iv: "38f50c2f96fc434ee8cafffd757f8086"
server_key_version: "encryptionkey@0"
I20260620 18:41:29.485113 5495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-0/wal/instance:
uuid: "f758242b8e3d4b2e8742c80587dbc12c"
format_stamp: "Formatted at 2026-06-20 18:41:29 on dist-test-slave-d645"
server_key: "e7283e95ed8c97ee2dd7a36dcede1350"
server_key_iv: "38f50c2f96fc434ee8cafffd757f8086"
server_key_version: "encryptionkey@0"
I20260620 18:41:29.490147 5495 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 18:41:29.493611 5510 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:29.495047 5495 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:41:29.495206 5495 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-0/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-0/wal
uuid: "f758242b8e3d4b2e8742c80587dbc12c"
format_stamp: "Formatted at 2026-06-20 18:41:29 on dist-test-slave-d645"
server_key: "e7283e95ed8c97ee2dd7a36dcede1350"
server_key_iv: "38f50c2f96fc434ee8cafffd757f8086"
server_key_version: "encryptionkey@0"
I20260620 18:41:29.495404 5495 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:41:29.515841 5495 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:41:29.520596 5495 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:41:29.520869 5495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:41:29.521926 5495 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:41:29.523579 5495 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:41:29.523661 5495 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:29.523789 5495 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:41:29.523838 5495 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:29.554205 5495 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.1:46031
I20260620 18:41:29.554240 5622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.1:46031 every 8 connection(s)
I20260620 18:41:29.555559 5495 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-0/data/info.pb
I20260620 18:41:29.564589 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 5495
I20260620 18:41:29.564803 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-0/wal/instance
I20260620 18:41:29.565415 24012 external_mini_cluster.cc:1428] Setting key cd0214bfc7a6bdc407fd8947e4f4397a
Jun 20 18:41:29 dist-test-slave-d645 krb5kdc[5389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980889, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.1@KRBTEST.COM for kudu/127.23.115.62@KRBTEST.COM
I20260620 18:41:29.574797 5625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:29.558060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.1:34483 (local address 127.23.115.62:35467)
0620 18:41:29.558512 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:29.558517 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:29.559424 (+ 907us) server_negotiation.cc:407] Connection header received
0620 18:41:29.561462 (+ 2038us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:29.561467 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:29.561549 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:29.561661 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:29.563068 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:29.563801 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:29.565408 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:29.565848 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:29.565958 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:29.569269 (+ 3311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:29.569275 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:29.569278 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:29.569311 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:29.571845 (+ 2534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:29.572594 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:29.572598 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:29.572601 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:29.572714 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:29.573130 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:29.573133 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:29.573135 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:29.573400 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:29.573533 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:29.574206 (+ 673us) server_negotiation.cc:299] Negotiation successful
0620 18:41:29.574426 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":214,"threads_started":1}
I20260620 18:41:29.575752 5626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:29.558550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:35467 (local address 127.23.115.1:34483)
0620 18:41:29.559284 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:29.559320 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:29.561203 (+ 1883us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:29.561890 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:29.561897 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:29.562373 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:41:29.562883 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:29.562900 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:29.563951 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:29.563955 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:29.565264 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:29.565277 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:29.566078 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:29.566086 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:29.566284 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:29.566991 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:41:29.567018 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:41:29.569052 (+ 2034us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:41:29.571982 (+ 2930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:29.571991 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:29.572008 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:29.572458 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:29.572814 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:29.572819 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:29.572823 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:29.573013 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:29.573521 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:41:29.573529 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:41:29.573847 (+ 318us) client_negotiation.cc:769] Sending connection context
0620 18:41:29.574093 (+ 246us) client_negotiation.cc:240] Negotiation successful
0620 18:41:29.574603 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":256,"threads_started":1}
I20260620 18:41:29.577212 5623 heartbeater.cc:347] Connected to a master server at 127.23.115.62:35467
I20260620 18:41:29.577491 5623 heartbeater.cc:464] Registering TS with master...
I20260620 18:41:29.578213 5623 heartbeater.cc:511] Master 127.23.115.62:35467 requested a full tablet report, sending...
I20260620 18:41:29.580183 5434 ts_manager.cc:194] Registered new tserver with Master: f758242b8e3d4b2e8742c80587dbc12c (127.23.115.1:46031)
I20260620 18:41:29.581593 5434 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.115.1@KRBTEST.COM'} at 127.23.115.1:34483
WARNING: no policy specified for kudu/127.23.115.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.115.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.115.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.2.keytab.
Entry for principal kudu/127.23.115.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.2.keytab.
I20260620 18:41:29.614764 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.115.2:0
--local_ip_for_outbound_sockets=127.23.115.2
--webserver_interface=127.23.115.2
--webserver_port=0
--tserver_master_addrs=127.23.115.62:35467
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.2.keytab
--principal=kudu/127.23.115.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:33195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:41:29.926894 5629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:41:29.927253 5629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:41:29.927361 5629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:41:29.934626 5629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:41:29.934732 5629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:41:29.934844 5629 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.115.2
I20260620 18:41:29.944571 5629 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:33195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.2.keytab
--principal=kudu/127.23.115.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.115.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.115.2
--webserver_port=0
--tserver_master_addrs=127.23.115.62:35467
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.115.2
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:41:29.945776 5629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:41:29.947001 5629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:41:29.959529 5635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:29.959563 5634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:29.960057 5637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:41:29.961308 5629 server_base.cc:1034] running on GCE node
I20260620 18:41:29.961896 5629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:41:29.962821 5629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:41:29.964023 5629 hybrid_clock.cc:648] HybridClock initialized: now 1781980889963961 us; error 60 us; skew 500 ppm
I20260620 18:41:29.964354 5629 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:41:29 dist-test-slave-d645 krb5kdc[5389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980889, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:41:29.967410 5629 init.cc:377] Logged in from keytab as kudu/127.23.115.2@KRBTEST.COM (short username kudu)
I20260620 18:41:29.968914 5629 webserver.cc:460] Webserver started at http://127.23.115.2:41981/ using document root <none> and password file <none>
I20260620 18:41:29.969614 5629 fs_manager.cc:359] Metadata directory not provided
I20260620 18:41:29.969756 5629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:41:29.970079 5629 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:41:29.973090 5629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-1/data/instance:
uuid: "f1e747c088b146c3a4eb96b53a980fb7"
format_stamp: "Formatted at 2026-06-20 18:41:29 on dist-test-slave-d645"
server_key: "c0a8216d3a3a482d534a45555a5a35b1"
server_key_iv: "bb1d71dc245e2bfd52b199f6f4c7a19d"
server_key_version: "encryptionkey@0"
I20260620 18:41:29.974040 5629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-1/wal/instance:
uuid: "f1e747c088b146c3a4eb96b53a980fb7"
format_stamp: "Formatted at 2026-06-20 18:41:29 on dist-test-slave-d645"
server_key: "c0a8216d3a3a482d534a45555a5a35b1"
server_key_iv: "bb1d71dc245e2bfd52b199f6f4c7a19d"
server_key_version: "encryptionkey@0"
I20260620 18:41:29.980362 5629 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260620 18:41:29.984289 5644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:29.985827 5629 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:41:29.985976 5629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-1/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-1/wal
uuid: "f1e747c088b146c3a4eb96b53a980fb7"
format_stamp: "Formatted at 2026-06-20 18:41:29 on dist-test-slave-d645"
server_key: "c0a8216d3a3a482d534a45555a5a35b1"
server_key_iv: "bb1d71dc245e2bfd52b199f6f4c7a19d"
server_key_version: "encryptionkey@0"
I20260620 18:41:29.986174 5629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:41:30.018213 5629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:41:30.024974 5629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:41:30.025627 5629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:41:30.027019 5629 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:41:30.028736 5629 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:41:30.028826 5629 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:30.028929 5629 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:41:30.028986 5629 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:30.060930 5629 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.2:36887
I20260620 18:41:30.060972 5756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.2:36887 every 8 connection(s)
I20260620 18:41:30.062122 5629 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-1/data/info.pb
I20260620 18:41:30.066570 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 5629
I20260620 18:41:30.066764 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-1/wal/instance
I20260620 18:41:30.067226 24012 external_mini_cluster.cc:1428] Setting key ea820b471010620779606f7f70701f9b
Jun 20 18:41:30 dist-test-slave-d645 krb5kdc[5389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980889, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.2@KRBTEST.COM for kudu/127.23.115.62@KRBTEST.COM
I20260620 18:41:30.081569 5625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:30.064577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.2:58745 (local address 127.23.115.62:35467)
0620 18:41:30.064756 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:30.064761 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:30.065885 (+ 1124us) server_negotiation.cc:407] Connection header received
0620 18:41:30.067938 (+ 2053us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:30.067944 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:30.068037 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:30.068156 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:30.069902 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:30.070591 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:30.072125 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:30.072551 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:30.072670 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:30.075968 (+ 3298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:30.075973 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:30.075977 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:30.076009 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:30.078483 (+ 2474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:30.079262 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:30.079266 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:30.079268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:30.079391 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:30.079818 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:30.079821 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:30.079823 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:30.080132 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:30.080265 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:30.080973 (+ 708us) server_negotiation.cc:299] Negotiation successful
0620 18:41:30.081225 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 18:41:30.082571 5759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:30.064953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:35467 (local address 127.23.115.2:58745)
0620 18:41:30.065716 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:30.065760 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:30.067626 (+ 1866us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:30.068402 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:30.068412 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:30.069115 (+ 703us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:41:30.069742 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:30.069760 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:30.070748 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:30.070752 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:30.071986 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:30.071998 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:30.072762 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:30.072769 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:30.072984 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:30.073599 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:41:30.073625 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:41:30.075748 (+ 2123us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:41:30.078624 (+ 2876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:30.078634 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:30.078652 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:30.079143 (+ 491us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:30.079494 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:30.079500 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:30.079503 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:30.079704 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:30.080254 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:41:30.080262 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:41:30.080600 (+ 338us) client_negotiation.cc:769] Sending connection context
0620 18:41:30.080862 (+ 262us) client_negotiation.cc:240] Negotiation successful
0620 18:41:30.081438 (+ 576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":290,"threads_started":1}
I20260620 18:41:30.084110 5757 heartbeater.cc:347] Connected to a master server at 127.23.115.62:35467
I20260620 18:41:30.084470 5757 heartbeater.cc:464] Registering TS with master...
I20260620 18:41:30.085216 5757 heartbeater.cc:511] Master 127.23.115.62:35467 requested a full tablet report, sending...
I20260620 18:41:30.086671 5434 ts_manager.cc:194] Registered new tserver with Master: f1e747c088b146c3a4eb96b53a980fb7 (127.23.115.2:36887)
I20260620 18:41:30.087474 5434 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.115.2@KRBTEST.COM'} at 127.23.115.2:58745
WARNING: no policy specified for kudu/127.23.115.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.115.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.115.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.3.keytab.
Entry for principal kudu/127.23.115.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.3.keytab.
I20260620 18:41:30.118736 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.115.3:0
--local_ip_for_outbound_sockets=127.23.115.3
--webserver_interface=127.23.115.3
--webserver_port=0
--tserver_master_addrs=127.23.115.62:35467
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.3.keytab
--principal=kudu/127.23.115.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:33195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:41:30.431282 5762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:41:30.431543 5762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:41:30.431658 5762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:41:30.438719 5762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:41:30.438822 5762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:41:30.439008 5762 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.115.3
I20260620 18:41:30.448491 5762 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:33195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.3.keytab
--principal=kudu/127.23.115.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.115.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.115.3
--webserver_port=0
--tserver_master_addrs=127.23.115.62:35467
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.115.3
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:41:30.449766 5762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:41:30.451012 5762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:41:30.463280 5767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:30.463225 5770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:30.463258 5768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:41:30.464295 5762 server_base.cc:1034] running on GCE node
I20260620 18:41:30.464829 5762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:41:30.465708 5762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:41:30.466920 5762 hybrid_clock.cc:648] HybridClock initialized: now 1781980890466875 us; error 33 us; skew 500 ppm
I20260620 18:41:30.467283 5762 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:41:30 dist-test-slave-d645 krb5kdc[5389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980890, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:41:30.470625 5762 init.cc:377] Logged in from keytab as kudu/127.23.115.3@KRBTEST.COM (short username kudu)
I20260620 18:41:30.472229 5762 webserver.cc:460] Webserver started at http://127.23.115.3:43223/ using document root <none> and password file <none>
I20260620 18:41:30.472882 5762 fs_manager.cc:359] Metadata directory not provided
I20260620 18:41:30.472994 5762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:41:30.473278 5762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:41:30.476184 5762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-2/data/instance:
uuid: "b32dcc4d34fc44908dd3d879251d0ef0"
format_stamp: "Formatted at 2026-06-20 18:41:30 on dist-test-slave-d645"
server_key: "3adba41137f35f38872966a1dc3cd1e6"
server_key_iv: "6b0c1a0c31be04487a2a6d029ab1eea9"
server_key_version: "encryptionkey@0"
I20260620 18:41:30.476938 5762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-2/wal/instance:
uuid: "b32dcc4d34fc44908dd3d879251d0ef0"
format_stamp: "Formatted at 2026-06-20 18:41:30 on dist-test-slave-d645"
server_key: "3adba41137f35f38872966a1dc3cd1e6"
server_key_iv: "6b0c1a0c31be04487a2a6d029ab1eea9"
server_key_version: "encryptionkey@0"
I20260620 18:41:30.481869 5762 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 18:41:30.485390 5777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:30.486734 5762 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260620 18:41:30.486898 5762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-2/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-2/wal
uuid: "b32dcc4d34fc44908dd3d879251d0ef0"
format_stamp: "Formatted at 2026-06-20 18:41:30 on dist-test-slave-d645"
server_key: "3adba41137f35f38872966a1dc3cd1e6"
server_key_iv: "6b0c1a0c31be04487a2a6d029ab1eea9"
server_key_version: "encryptionkey@0"
I20260620 18:41:30.487150 5762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:41:30.505878 5762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:41:30.509820 5762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:41:30.510110 5762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:41:30.511080 5762 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:41:30.512382 5762 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:41:30.512450 5762 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:30.512543 5762 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:41:30.512584 5762 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:30.543452 5762 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.3:41761
I20260620 18:41:30.543541 5889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.3:41761 every 8 connection(s)
I20260620 18:41:30.544652 5762 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-2/data/info.pb
I20260620 18:41:30.551272 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 5762
I20260620 18:41:30.551474 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-2/wal/instance
I20260620 18:41:30.552059 24012 external_mini_cluster.cc:1428] Setting key 10f18e3b1dd97512ad034c8bf616fbcc
Jun 20 18:41:30 dist-test-slave-d645 krb5kdc[5389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980890, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.3@KRBTEST.COM for kudu/127.23.115.62@KRBTEST.COM
I20260620 18:41:30.565510 5625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:30.547133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.3:34703 (local address 127.23.115.62:35467)
0620 18:41:30.547306 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:30.547310 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:30.548413 (+ 1103us) server_negotiation.cc:407] Connection header received
0620 18:41:30.550253 (+ 1840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:30.550259 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:30.550375 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:30.550477 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:30.552392 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:30.553337 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:30.555630 (+ 2293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:30.556001 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:30.556101 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:30.560082 (+ 3981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:30.560087 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:30.560091 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:30.560123 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:30.562527 (+ 2404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:30.563249 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:30.563253 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:30.563255 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:30.563383 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:30.563850 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:30.563854 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:30.563856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:30.564134 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:30.564264 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:30.564940 (+ 676us) server_negotiation.cc:299] Negotiation successful
0620 18:41:30.565176 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 18:41:30.566471 5892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:30.547552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:35467 (local address 127.23.115.3:34703)
0620 18:41:30.548259 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:30.548298 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:30.549987 (+ 1689us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:30.550715 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:30.550723 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:30.551458 (+ 735us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:41:30.552205 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:30.552233 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:30.553544 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:30.553549 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:30.555468 (+ 1919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:30.555483 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:30.556221 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:30.556229 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:30.556503 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:30.557256 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:41:30.557292 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:41:30.559865 (+ 2573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:41:30.562657 (+ 2792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:30.562666 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:30.562680 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:30.563129 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:30.563486 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:30.563491 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:30.563495 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:30.563740 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:30.564253 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:41:30.564260 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:41:30.564575 (+ 315us) client_negotiation.cc:769] Sending connection context
0620 18:41:30.564819 (+ 244us) client_negotiation.cc:240] Negotiation successful
0620 18:41:30.565361 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":269,"threads_started":1}
I20260620 18:41:30.567833 5890 heartbeater.cc:347] Connected to a master server at 127.23.115.62:35467
I20260620 18:41:30.568122 5890 heartbeater.cc:464] Registering TS with master...
I20260620 18:41:30.568784 5890 heartbeater.cc:511] Master 127.23.115.62:35467 requested a full tablet report, sending...
I20260620 18:41:30.570158 5434 ts_manager.cc:194] Registered new tserver with Master: b32dcc4d34fc44908dd3d879251d0ef0 (127.23.115.3:41761)
I20260620 18:41:30.570919 5434 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.115.3@KRBTEST.COM'} at 127.23.115.3:34703
I20260620 18:41:30.579341 24012 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:41:30.584514 5623 heartbeater.cc:503] Master 127.23.115.62:35467 was elected leader, sending a full tablet report...
I20260620 18:41:30.597076 5625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:30.583796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41330 (local address 127.23.115.62:35467)
0620 18:41:30.583985 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:30.583990 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:30.584403 (+ 413us) server_negotiation.cc:407] Connection header received
0620 18:41:30.584534 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:30.584540 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:30.584622 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:30.584745 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:30.586398 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:30.587185 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:30.588569 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:30.588994 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:30.589092 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:30.591675 (+ 2583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:30.591680 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:30.591683 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:30.591742 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:30.594143 (+ 2401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:30.594905 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:30.594910 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:30.594912 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:30.595123 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:30.595480 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:30.595483 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:30.595485 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:30.595801 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:30.595988 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:30.596628 (+ 640us) server_negotiation.cc:299] Negotiation successful
0620 18:41:30.596799 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 18:41:30.601512 5434 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41330:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:41:30.608315 5434 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:41:30.629679 5903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:30.620407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34658 (local address 127.23.115.2:36887)
0620 18:41:30.620985 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:30.620994 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:30.621118 (+ 124us) server_negotiation.cc:407] Connection header received
0620 18:41:30.621528 (+ 410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:30.621536 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:30.621770 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:30.622014 (+ 244us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:41:30.623143 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:30.624648 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:30.627244 (+ 2596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:30.628570 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:30.628709 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:30.628906 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:30.629200 (+ 294us) server_negotiation.cc:299] Negotiation successful
0620 18:41:30.629368 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":310,"threads_started":1}
I20260620 18:41:30.629896 5902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:30.620141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.2:36887 (local address 127.0.0.1:34658)
0620 18:41:30.620921 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:30.621025 (+ 104us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:30.621237 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:30.622225 (+ 988us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:30.622233 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:30.622268 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:41:30.622785 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:30.622798 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:30.624825 (+ 2027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:30.624830 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:30.627056 (+ 2226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:30.627074 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:30.628783 (+ 1709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:30.628789 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:30.628965 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:30.629014 (+ 49us) client_negotiation.cc:769] Sending connection context
0620 18:41:30.629121 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 18:41:30.629311 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408,"mutex_wait_us":32,"thread_start_us":279,"threads_started":1}
I20260620 18:41:30.631399 5905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:30.621568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50404 (local address 127.23.115.3:41761)
0620 18:41:30.623269 (+ 1701us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:30.623277 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:30.623307 (+ 30us) server_negotiation.cc:407] Connection header received
0620 18:41:30.623395 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:30.623401 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:30.623600 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:30.623776 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:41:30.624462 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:30.626007 (+ 1545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:30.628253 (+ 2246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:30.630077 (+ 1824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:30.630214 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:30.630431 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:30.631010 (+ 579us) server_negotiation.cc:299] Negotiation successful
0620 18:41:30.631155 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1530,"thread_start_us":1349,"threads_started":1}
I20260620 18:41:30.634479 5906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:30.622906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.1:46031 (local address 127.0.0.1:33330)
0620 18:41:30.623473 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:30.623491 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:30.623595 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:30.627157 (+ 3562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:30.627161 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:30.627187 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:41:30.627541 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:30.627552 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:30.629682 (+ 2130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:30.629692 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:30.632090 (+ 2398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:30.632106 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:30.633932 (+ 1826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:30.633937 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:30.634062 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:30.634087 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 18:41:30.634164 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 18:41:30.634253 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":780,"threads_started":1}
I20260620 18:41:30.633867 5692 tablet_service.cc:1467] Processing CreateTablet for tablet d0e1857395724258adfc1085b5432e4c (DEFAULT_TABLE table=test-table [id=3d4626fc11314092a7aa2ac05c123436]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:41:30.634684 5907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:30.623529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33330 (local address 127.23.115.1:46031)
0620 18:41:30.626590 (+ 3061us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:30.626598 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:30.626622 (+ 24us) server_negotiation.cc:407] Connection header received
0620 18:41:30.626715 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:30.626722 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:30.626968 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:30.627155 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:41:30.627752 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:30.629232 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:30.632307 (+ 3075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:30.633704 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:30.633847 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:30.634071 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:30.634221 (+ 150us) server_negotiation.cc:299] Negotiation successful
0620 18:41:30.634374 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2895,"thread_start_us":2754,"threads_started":1}
I20260620 18:41:30.635252 5904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:30.621330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.3:41761 (local address 127.0.0.1:50404)
0620 18:41:30.622320 (+ 990us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:30.622340 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:30.622515 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:30.623876 (+ 1361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:30.623880 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:30.623904 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:41:30.624250 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:30.624261 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:30.626145 (+ 1884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:30.626151 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:30.628063 (+ 1912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:30.628076 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:30.630599 (+ 2523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:30.630606 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:30.630786 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:30.630826 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 18:41:30.630923 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 18:41:30.634871 (+ 3948us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":877,"mutex_wait_us":2,"spinlock_wait_cycles":10624,"thread_start_us":720,"threads_started":1}
I20260620 18:41:30.635371 5692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0e1857395724258adfc1085b5432e4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:41:30.638396 5825 tablet_service.cc:1467] Processing CreateTablet for tablet d0e1857395724258adfc1085b5432e4c (DEFAULT_TABLE table=test-table [id=3d4626fc11314092a7aa2ac05c123436]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:41:30.638442 5558 tablet_service.cc:1467] Processing CreateTablet for tablet d0e1857395724258adfc1085b5432e4c (DEFAULT_TABLE table=test-table [id=3d4626fc11314092a7aa2ac05c123436]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:41:30.639592 5825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0e1857395724258adfc1085b5432e4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:41:30.639775 5558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0e1857395724258adfc1085b5432e4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:41:30.652240 5909 tablet_bootstrap.cc:492] T d0e1857395724258adfc1085b5432e4c P f1e747c088b146c3a4eb96b53a980fb7: Bootstrap starting.
I20260620 18:41:30.653093 5908 tablet_bootstrap.cc:492] T d0e1857395724258adfc1085b5432e4c P b32dcc4d34fc44908dd3d879251d0ef0: Bootstrap starting.
I20260620 18:41:30.654495 5910 tablet_bootstrap.cc:492] T d0e1857395724258adfc1085b5432e4c P f758242b8e3d4b2e8742c80587dbc12c: Bootstrap starting.
I20260620 18:41:30.656059 5908 tablet_bootstrap.cc:654] T d0e1857395724258adfc1085b5432e4c P b32dcc4d34fc44908dd3d879251d0ef0: Neither blocks nor log segments found. Creating new log.
I20260620 18:41:30.656111 5909 tablet_bootstrap.cc:654] T d0e1857395724258adfc1085b5432e4c P f1e747c088b146c3a4eb96b53a980fb7: Neither blocks nor log segments found. Creating new log.
I20260620 18:41:30.657039 5908 log.cc:824] T d0e1857395724258adfc1085b5432e4c P b32dcc4d34fc44908dd3d879251d0ef0: Log is configured to *not* fsync() on all Append() calls
I20260620 18:41:30.657698 5910 tablet_bootstrap.cc:654] T d0e1857395724258adfc1085b5432e4c P f758242b8e3d4b2e8742c80587dbc12c: Neither blocks nor log segments found. Creating new log.
I20260620 18:41:30.657938 5909 log.cc:824] T d0e1857395724258adfc1085b5432e4c P f1e747c088b146c3a4eb96b53a980fb7: Log is configured to *not* fsync() on all Append() calls
I20260620 18:41:30.658752 5910 log.cc:824] T d0e1857395724258adfc1085b5432e4c P f758242b8e3d4b2e8742c80587dbc12c: Log is configured to *not* fsync() on all Append() calls
I20260620 18:41:30.660604 5909 tablet_bootstrap.cc:492] T d0e1857395724258adfc1085b5432e4c P f1e747c088b146c3a4eb96b53a980fb7: No bootstrap required, opened a new log
I20260620 18:41:30.660815 5908 tablet_bootstrap.cc:492] T d0e1857395724258adfc1085b5432e4c P b32dcc4d34fc44908dd3d879251d0ef0: No bootstrap required, opened a new log
I20260620 18:41:30.660890 5909 ts_tablet_manager.cc:1397] T d0e1857395724258adfc1085b5432e4c P f1e747c088b146c3a4eb96b53a980fb7: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260620 18:41:30.661091 5908 ts_tablet_manager.cc:1397] T d0e1857395724258adfc1085b5432e4c P b32dcc4d34fc44908dd3d879251d0ef0: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260620 18:41:30.667783 5909 raft_consensus.cc:348] T d0e1857395724258adfc1085b5432e4c P f1e747c088b146c3a4eb96b53a980fb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e747c088b146c3a4eb96b53a980fb7" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 36887 } } peers { permanent_uuid: "b32dcc4d34fc44908dd3d879251d0ef0" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 41761 } } peers { permanent_uuid: "f758242b8e3d4b2e8742c80587dbc12c" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 46031 } }
I20260620 18:41:30.668334 5909 raft_consensus.cc:374] T d0e1857395724258adfc1085b5432e4c P f1e747c088b146c3a4eb96b53a980fb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:41:30.668427 5909 raft_consensus.cc:729] T d0e1857395724258adfc1085b5432e4c P f1e747c088b146c3a4eb96b53a980fb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e747c088b146c3a4eb96b53a980fb7, State: Initialized, Role: FOLLOWER
I20260620 18:41:30.668979 5909 consensus_queue.cc:260] T d0e1857395724258adfc1085b5432e4c P f1e747c088b146c3a4eb96b53a980fb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e747c088b146c3a4eb96b53a980fb7" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 36887 } } peers { permanent_uuid: "b32dcc4d34fc44908dd3d879251d0ef0" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 41761 } } peers { permanent_uuid: "f758242b8e3d4b2e8742c80587dbc12c" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 46031 } }
I20260620 18:41:30.669659 5910 tablet_bootstrap.cc:492] T d0e1857395724258adfc1085b5432e4c P f758242b8e3d4b2e8742c80587dbc12c: No bootstrap required, opened a new log
I20260620 18:41:30.669976 5910 ts_tablet_manager.cc:1397] T d0e1857395724258adfc1085b5432e4c P f758242b8e3d4b2e8742c80587dbc12c: Time spent bootstrapping tablet: real 0.016s user 0.011s sys 0.004s
I20260620 18:41:30.669996 5757 heartbeater.cc:503] Master 127.23.115.62:35467 was elected leader, sending a full tablet report...
I20260620 18:41:30.670608 5909 ts_tablet_manager.cc:1428] T d0e1857395724258adfc1085b5432e4c P f1e747c088b146c3a4eb96b53a980fb7: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20260620 18:41:30.671754 5908 raft_consensus.cc:348] T d0e1857395724258adfc1085b5432e4c P b32dcc4d34fc44908dd3d879251d0ef0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e747c088b146c3a4eb96b53a980fb7" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 36887 } } peers { permanent_uuid: "b32dcc4d34fc44908dd3d879251d0ef0" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 41761 } } peers { permanent_uuid: "f758242b8e3d4b2e8742c80587dbc12c" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 46031 } }
I20260620 18:41:30.672214 5908 raft_consensus.cc:374] T d0e1857395724258adfc1085b5432e4c P b32dcc4d34fc44908dd3d879251d0ef0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:41:30.672348 5908 raft_consensus.cc:729] T d0e1857395724258adfc1085b5432e4c P b32dcc4d34fc44908dd3d879251d0ef0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b32dcc4d34fc44908dd3d879251d0ef0, State: Initialized, Role: FOLLOWER
I20260620 18:41:30.673053 5908 consensus_queue.cc:260] T d0e1857395724258adfc1085b5432e4c P b32dcc4d34fc44908dd3d879251d0ef0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e747c088b146c3a4eb96b53a980fb7" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 36887 } } peers { permanent_uuid: "b32dcc4d34fc44908dd3d879251d0ef0" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 41761 } } peers { permanent_uuid: "f758242b8e3d4b2e8742c80587dbc12c" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 46031 } }
I20260620 18:41:30.674232 5890 heartbeater.cc:503] Master 127.23.115.62:35467 was elected leader, sending a full tablet report...
I20260620 18:41:30.674814 5908 ts_tablet_manager.cc:1428] T d0e1857395724258adfc1085b5432e4c P b32dcc4d34fc44908dd3d879251d0ef0: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260620 18:41:30.679493 5910 raft_consensus.cc:348] T d0e1857395724258adfc1085b5432e4c P f758242b8e3d4b2e8742c80587dbc12c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e747c088b146c3a4eb96b53a980fb7" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 36887 } } peers { permanent_uuid: "b32dcc4d34fc44908dd3d879251d0ef0" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 41761 } } peers { permanent_uuid: "f758242b8e3d4b2e8742c80587dbc12c" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 46031 } }
I20260620 18:41:30.679839 5910 raft_consensus.cc:374] T d0e1857395724258adfc1085b5432e4c P f758242b8e3d4b2e8742c80587dbc12c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:41:30.679932 5910 raft_consensus.cc:729] T d0e1857395724258adfc1085b5432e4c P f758242b8e3d4b2e8742c80587dbc12c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f758242b8e3d4b2e8742c80587dbc12c, State: Initialized, Role: FOLLOWER
I20260620 18:41:30.680457 5910 consensus_queue.cc:260] T d0e1857395724258adfc1085b5432e4c P f758242b8e3d4b2e8742c80587dbc12c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e747c088b146c3a4eb96b53a980fb7" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 36887 } } peers { permanent_uuid: "b32dcc4d34fc44908dd3d879251d0ef0" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 41761 } } peers { permanent_uuid: "f758242b8e3d4b2e8742c80587dbc12c" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 46031 } }
I20260620 18:41:30.682067 5910 ts_tablet_manager.cc:1428] T d0e1857395724258adfc1085b5432e4c P f758242b8e3d4b2e8742c80587dbc12c: Time spent starting tablet: real 0.012s user 0.007s sys 0.004s
I20260620 18:41:30.757001 5914 raft_consensus.cc:482] T d0e1857395724258adfc1085b5432e4c P f1e747c088b146c3a4eb96b53a980fb7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:41:30.757296 5914 raft_consensus.cc:504] T d0e1857395724258adfc1085b5432e4c P f1e747c088b146c3a4eb96b53a980fb7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e747c088b146c3a4eb96b53a980fb7" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 36887 } } peers { permanent_uuid: "b32dcc4d34fc44908dd3d879251d0ef0" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 41761 } } peers { permanent_uuid: "f758242b8e3d4b2e8742c80587dbc12c" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 46031 } }
I20260620 18:41:30.758908 5914 leader_election.cc:288] T d0e1857395724258adfc1085b5432e4c P f1e747c088b146c3a4eb96b53a980fb7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b32dcc4d34fc44908dd3d879251d0ef0 (127.23.115.3:41761), f758242b8e3d4b2e8742c80587dbc12c (127.23.115.1:46031)
I20260620 18:41:30.765929 5905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:30.759390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.2:40253 (local address 127.23.115.3:41761)
0620 18:41:30.759566 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:30.759570 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:30.759993 (+ 423us) server_negotiation.cc:407] Connection header received
0620 18:41:30.760184 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:30.760189 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:30.760271 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:30.760377 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:41:30.761055 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:30.762016 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:30.763989 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:30.764963 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:30.765089 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:30.765143 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:30.765532 (+ 389us) server_negotiation.cc:299] Negotiation successful
0620 18:41:30.765683 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 18:41:30.765995 5917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:30.759237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.3:41761 (local address 127.23.115.2:40253)
0620 18:41:30.759867 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:30.759886 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:30.760029 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:30.760463 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:30.760468 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:30.760495 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:41:30.760855 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:30.760867 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:30.762144 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:30.762150 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:30.763819 (+ 1669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:30.763834 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:30.765179 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:30.765186 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:30.765362 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:30.765398 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 18:41:30.765502 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 18:41:30.765647 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"mutex_wait_us":5,"thread_start_us":307,"threads_started":1}
I20260620 18:41:30.766690 5918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:30.759943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.1:46031 (local address 127.23.115.2:54423)
0620 18:41:30.760552 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:30.760586 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:30.760687 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:30.761050 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:30.761055 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:30.761086 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:41:30.761501 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:30.761513 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:30.762710 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:30.762714 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:30.764326 (+ 1612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:30.764340 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:30.765625 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:30.765631 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:30.765825 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:30.765858 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 18:41:30.765960 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 18:41:30.766089 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":336,"threads_started":1}
I20260620 18:41:30.766891 5907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:30.760168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.2:54423 (local address 127.23.115.1:46031)
0620 18:41:30.760371 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:30.760377 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:30.760668 (+ 291us) server_negotiation.cc:407] Connection header received
0620 18:41:30.760793 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:30.760798 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:30.760870 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:30.760959 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:41:30.761693 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:30.762546 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:30.764485 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:30.765422 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:30.765558 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:30.765623 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:30.765992 (+ 369us) server_negotiation.cc:299] Negotiation successful
0620 18:41:30.766121 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 18:41:30.767643 5845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0e1857395724258adfc1085b5432e4c" candidate_uuid: "f1e747c088b146c3a4eb96b53a980fb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b32dcc4d34fc44908dd3d879251d0ef0" is_pre_election: true
I20260620 18:41:30.767838 5578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0e1857395724258adfc1085b5432e4c" candidate_uuid: "f1e747c088b146c3a4eb96b53a980fb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f758242b8e3d4b2e8742c80587dbc12c" is_pre_election: true
I20260620 18:41:30.768251 5845 raft_consensus.cc:2454] T d0e1857395724258adfc1085b5432e4c P b32dcc4d34fc44908dd3d879251d0ef0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e747c088b146c3a4eb96b53a980fb7 in term 0.
I20260620 18:41:30.768352 5578 raft_consensus.cc:2454] T d0e1857395724258adfc1085b5432e4c P f758242b8e3d4b2e8742c80587dbc12c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e747c088b146c3a4eb96b53a980fb7 in term 0.
I20260620 18:41:30.768988 5648 leader_election.cc:302] T d0e1857395724258adfc1085b5432e4c P f1e747c088b146c3a4eb96b53a980fb7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f1e747c088b146c3a4eb96b53a980fb7, f758242b8e3d4b2e8742c80587dbc12c; no voters:
I20260620 18:41:30.769433 5914 raft_consensus.cc:2781] T d0e1857395724258adfc1085b5432e4c P f1e747c088b146c3a4eb96b53a980fb7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:41:30.769585 5914 raft_consensus.cc:482] T d0e1857395724258adfc1085b5432e4c P f1e747c088b146c3a4eb96b53a980fb7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:41:30.769692 5914 raft_consensus.cc:3037] T d0e1857395724258adfc1085b5432e4c P f1e747c088b146c3a4eb96b53a980fb7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:41:30.771881 5914 raft_consensus.cc:504] T d0e1857395724258adfc1085b5432e4c P f1e747c088b146c3a4eb96b53a980fb7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e747c088b146c3a4eb96b53a980fb7" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 36887 } } peers { permanent_uuid: "b32dcc4d34fc44908dd3d879251d0ef0" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 41761 } } peers { permanent_uuid: "f758242b8e3d4b2e8742c80587dbc12c" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 46031 } }
I20260620 18:41:30.772455 5914 leader_election.cc:288] T d0e1857395724258adfc1085b5432e4c P f1e747c088b146c3a4eb96b53a980fb7 [CANDIDATE]: Term 1 election: Requested vote from peers b32dcc4d34fc44908dd3d879251d0ef0 (127.23.115.3:41761), f758242b8e3d4b2e8742c80587dbc12c (127.23.115.1:46031)
I20260620 18:41:30.772926 5845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0e1857395724258adfc1085b5432e4c" candidate_uuid: "f1e747c088b146c3a4eb96b53a980fb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b32dcc4d34fc44908dd3d879251d0ef0"
I20260620 18:41:30.772967 5578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0e1857395724258adfc1085b5432e4c" candidate_uuid: "f1e747c088b146c3a4eb96b53a980fb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f758242b8e3d4b2e8742c80587dbc12c"
I20260620 18:41:30.773115 5845 raft_consensus.cc:3037] T d0e1857395724258adfc1085b5432e4c P b32dcc4d34fc44908dd3d879251d0ef0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:41:30.773164 5578 raft_consensus.cc:3037] T d0e1857395724258adfc1085b5432e4c P f758242b8e3d4b2e8742c80587dbc12c [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:41:30.775352 5845 raft_consensus.cc:2454] T d0e1857395724258adfc1085b5432e4c P b32dcc4d34fc44908dd3d879251d0ef0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e747c088b146c3a4eb96b53a980fb7 in term 1.
I20260620 18:41:30.775368 5578 raft_consensus.cc:2454] T d0e1857395724258adfc1085b5432e4c P f758242b8e3d4b2e8742c80587dbc12c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e747c088b146c3a4eb96b53a980fb7 in term 1.
I20260620 18:41:30.775729 5648 leader_election.cc:302] T d0e1857395724258adfc1085b5432e4c P f1e747c088b146c3a4eb96b53a980fb7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b32dcc4d34fc44908dd3d879251d0ef0, f1e747c088b146c3a4eb96b53a980fb7; no voters:
I20260620 18:41:30.776019 5914 raft_consensus.cc:2781] T d0e1857395724258adfc1085b5432e4c P f1e747c088b146c3a4eb96b53a980fb7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:41:30.776477 5914 raft_consensus.cc:686] T d0e1857395724258adfc1085b5432e4c P f1e747c088b146c3a4eb96b53a980fb7 [term 1 LEADER]: Becoming Leader. State: Replica: f1e747c088b146c3a4eb96b53a980fb7, State: Running, Role: LEADER
I20260620 18:41:30.776913 5914 consensus_queue.cc:237] T d0e1857395724258adfc1085b5432e4c P f1e747c088b146c3a4eb96b53a980fb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e747c088b146c3a4eb96b53a980fb7" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 36887 } } peers { permanent_uuid: "b32dcc4d34fc44908dd3d879251d0ef0" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 41761 } } peers { permanent_uuid: "f758242b8e3d4b2e8742c80587dbc12c" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 46031 } }
I20260620 18:41:30.781136 5433 catalog_manager.cc:5495] T d0e1857395724258adfc1085b5432e4c P f1e747c088b146c3a4eb96b53a980fb7 reported cstate change: term changed from 0 to 1, leader changed from <none> to f1e747c088b146c3a4eb96b53a980fb7 (127.23.115.2). New cstate: current_term: 1 leader_uuid: "f1e747c088b146c3a4eb96b53a980fb7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e747c088b146c3a4eb96b53a980fb7" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 36887 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b32dcc4d34fc44908dd3d879251d0ef0" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 41761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f758242b8e3d4b2e8742c80587dbc12c" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 46031 } health_report { overall_health: UNKNOWN } } }
W20260620 18:41:30.796238 5891 tablet.cc:2369] T d0e1857395724258adfc1085b5432e4c P b32dcc4d34fc44908dd3d879251d0ef0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:41:30.807874 5624 tablet.cc:2369] T d0e1857395724258adfc1085b5432e4c P f758242b8e3d4b2e8742c80587dbc12c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:41:30.814059 5758 tablet.cc:2369] T d0e1857395724258adfc1085b5432e4c P f1e747c088b146c3a4eb96b53a980fb7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:41:30.827220 5903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:30.821220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34662 (local address 127.23.115.2:36887)
0620 18:41:30.821442 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:30.821447 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:30.821465 (+ 18us) server_negotiation.cc:407] Connection header received
0620 18:41:30.821658 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:30.821662 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:30.821742 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:30.821861 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:41:30.822331 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:30.823181 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:30.824729 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:30.825291 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:30.825398 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:30.826006 (+ 608us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:41:30.826530 (+ 524us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:41:30.826647 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:30.826860 (+ 213us) server_negotiation.cc:299] Negotiation successful
0620 18:41:30.826978 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 18:41:30.835659 5845 raft_consensus.cc:1261] T d0e1857395724258adfc1085b5432e4c P b32dcc4d34fc44908dd3d879251d0ef0 [term 1 FOLLOWER]: Refusing update from remote peer f1e747c088b146c3a4eb96b53a980fb7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:41:30.835700 5578 raft_consensus.cc:1261] T d0e1857395724258adfc1085b5432e4c P f758242b8e3d4b2e8742c80587dbc12c [term 1 FOLLOWER]: Refusing update from remote peer f1e747c088b146c3a4eb96b53a980fb7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:41:30.836714 5914 consensus_queue.cc:1035] T d0e1857395724258adfc1085b5432e4c P f1e747c088b146c3a4eb96b53a980fb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f758242b8e3d4b2e8742c80587dbc12c" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 46031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:41:30.837054 5919 consensus_queue.cc:1035] T d0e1857395724258adfc1085b5432e4c P f1e747c088b146c3a4eb96b53a980fb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b32dcc4d34fc44908dd3d879251d0ef0" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 41761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:41:30.862861 5434 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41330:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:41:30.863211 5434 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41330:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:41:30.866506 5434 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a15618092ac84abc82a0f318082d7455: Sending DeleteTablet for 3 replicas of tablet d0e1857395724258adfc1085b5432e4c
I20260620 18:41:30.867630 5692 tablet_service.cc:1514] Processing DeleteTablet for tablet d0e1857395724258adfc1085b5432e4c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:41:30 UTC) from {username='kudu', principal='kudu/127.23.115.62@KRBTEST.COM'} at 127.0.0.1:34658
I20260620 18:41:30.867770 5825 tablet_service.cc:1514] Processing DeleteTablet for tablet d0e1857395724258adfc1085b5432e4c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:41:30 UTC) from {username='kudu', principal='kudu/127.23.115.62@KRBTEST.COM'} at 127.0.0.1:50404
I20260620 18:41:30.868186 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 5495
I20260620 18:41:30.868063 5558 tablet_service.cc:1514] Processing DeleteTablet for tablet d0e1857395724258adfc1085b5432e4c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:41:30 UTC) from {username='kudu', principal='kudu/127.23.115.62@KRBTEST.COM'} at 127.0.0.1:33330
I20260620 18:41:30.878796 5933 tablet_replica.cc:330] T d0e1857395724258adfc1085b5432e4c P b32dcc4d34fc44908dd3d879251d0ef0: stopping tablet replica
I20260620 18:41:30.879484 5933 raft_consensus.cc:2229] T d0e1857395724258adfc1085b5432e4c P b32dcc4d34fc44908dd3d879251d0ef0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:41:30.880126 5933 raft_consensus.cc:2258] T d0e1857395724258adfc1085b5432e4c P b32dcc4d34fc44908dd3d879251d0ef0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:41:30.880720 5932 tablet_replica.cc:330] T d0e1857395724258adfc1085b5432e4c P f1e747c088b146c3a4eb96b53a980fb7: stopping tablet replica
I20260620 18:41:30.881088 5934 tablet_replica.cc:330] T d0e1857395724258adfc1085b5432e4c P f758242b8e3d4b2e8742c80587dbc12c: stopping tablet replica
I20260620 18:41:30.881518 5932 raft_consensus.cc:2229] T d0e1857395724258adfc1085b5432e4c P f1e747c088b146c3a4eb96b53a980fb7 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:41:30.881570 5934 raft_consensus.cc:2229] T d0e1857395724258adfc1085b5432e4c P f758242b8e3d4b2e8742c80587dbc12c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:41:30.882257 5934 raft_consensus.cc:2258] T d0e1857395724258adfc1085b5432e4c P f758242b8e3d4b2e8742c80587dbc12c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:41:30.882346 5932 raft_consensus.cc:2258] T d0e1857395724258adfc1085b5432e4c P f1e747c088b146c3a4eb96b53a980fb7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:41:30.884316 5933 ts_tablet_manager.cc:1905] T d0e1857395724258adfc1085b5432e4c P b32dcc4d34fc44908dd3d879251d0ef0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:41:30.884344 5934 ts_tablet_manager.cc:1905] T d0e1857395724258adfc1085b5432e4c P f758242b8e3d4b2e8742c80587dbc12c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:41:30.891079 5932 ts_tablet_manager.cc:1905] T d0e1857395724258adfc1085b5432e4c P f1e747c088b146c3a4eb96b53a980fb7: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 20 18:41:30 dist-test-slave-d645 krb5kdc[5389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980888, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.115.1@KRBTEST.COM
I20260620 18:41:30.891702 5934 ts_tablet_manager.cc:1918] T d0e1857395724258adfc1085b5432e4c P f758242b8e3d4b2e8742c80587dbc12c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:41:30.891983 5934 log.cc:1196] T d0e1857395724258adfc1085b5432e4c P f758242b8e3d4b2e8742c80587dbc12c: Deleting WAL directory at /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-0/wal/wals/d0e1857395724258adfc1085b5432e4c
I20260620 18:41:30.892577 5934 ts_tablet_manager.cc:1939] T d0e1857395724258adfc1085b5432e4c P f758242b8e3d4b2e8742c80587dbc12c: Deleting consensus metadata
I20260620 18:41:30.895184 5421 catalog_manager.cc:4841] TS f758242b8e3d4b2e8742c80587dbc12c (127.23.115.1:46031): tablet d0e1857395724258adfc1085b5432e4c (table test-table [id=3d4626fc11314092a7aa2ac05c123436]) successfully deleted
I20260620 18:41:30.896023 5933 ts_tablet_manager.cc:1918] T d0e1857395724258adfc1085b5432e4c P b32dcc4d34fc44908dd3d879251d0ef0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:41:30.896217 5933 log.cc:1196] T d0e1857395724258adfc1085b5432e4c P b32dcc4d34fc44908dd3d879251d0ef0: Deleting WAL directory at /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-2/wal/wals/d0e1857395724258adfc1085b5432e4c
I20260620 18:41:30.896761 5933 ts_tablet_manager.cc:1939] T d0e1857395724258adfc1085b5432e4c P b32dcc4d34fc44908dd3d879251d0ef0: Deleting consensus metadata
I20260620 18:41:30.898502 5421 catalog_manager.cc:4841] TS b32dcc4d34fc44908dd3d879251d0ef0 (127.23.115.3:41761): tablet d0e1857395724258adfc1085b5432e4c (table test-table [id=3d4626fc11314092a7aa2ac05c123436]) successfully deleted
I20260620 18:41:30.898669 5932 ts_tablet_manager.cc:1918] T d0e1857395724258adfc1085b5432e4c P f1e747c088b146c3a4eb96b53a980fb7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:41:30.898849 5932 log.cc:1196] T d0e1857395724258adfc1085b5432e4c P f1e747c088b146c3a4eb96b53a980fb7: Deleting WAL directory at /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-1/wal/wals/d0e1857395724258adfc1085b5432e4c
I20260620 18:41:30.899425 5932 ts_tablet_manager.cc:1939] T d0e1857395724258adfc1085b5432e4c P f1e747c088b146c3a4eb96b53a980fb7: Deleting consensus metadata
I20260620 18:41:30.900864 5419 catalog_manager.cc:4841] TS f1e747c088b146c3a4eb96b53a980fb7 (127.23.115.2:36887): tablet d0e1857395724258adfc1085b5432e4c (table test-table [id=3d4626fc11314092a7aa2ac05c123436]) successfully deleted
I20260620 18:41:30.901612 5907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:30.869361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33338 (local address 127.23.115.1:46031)
0620 18:41:30.869553 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:30.869560 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:30.880396 (+ 10836us) server_negotiation.cc:407] Connection header received
0620 18:41:30.880947 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:30.880953 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:30.881062 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:30.881143 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:30.883210 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:30.884281 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:30.886223 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:30.886880 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:30.887012 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:30.892774 (+ 5762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:30.892789 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:30.892807 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:30.892850 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:30.897430 (+ 4580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:30.898894 (+ 1464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:30.898903 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:30.898907 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:30.899107 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:30.899676 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:30.899682 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:30.899694 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:30.900170 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:30.900497 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:30.900995 (+ 498us) server_negotiation.cc:299] Negotiation successful
0620 18:41:30.901241 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 18:41:30.902311 5618 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:41:31.020340 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 5495
I20260620 18:41:31.035712 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 5629
Jun 20 18:41:31 dist-test-slave-d645 krb5kdc[5389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980888, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.115.2@KRBTEST.COM
I20260620 18:41:31.053364 5903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:31.036444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34672 (local address 127.23.115.2:36887)
0620 18:41:31.036639 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:31.036645 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:31.036667 (+ 22us) server_negotiation.cc:407] Connection header received
0620 18:41:31.036777 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:31.036783 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:31.036873 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:31.036982 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:31.038202 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:31.039447 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:31.041135 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:31.041820 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:31.041957 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:31.047067 (+ 5110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:31.047084 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:31.047109 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:31.047157 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:31.050089 (+ 2932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:31.051011 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:31.051020 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:31.051024 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:31.051176 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:31.051619 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:31.051623 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:31.051626 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:31.052013 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:31.052309 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:31.052835 (+ 526us) server_negotiation.cc:299] Negotiation successful
0620 18:41:31.053054 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 18:41:31.054277 5752 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:41:31.166410 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 5629
I20260620 18:41:31.182449 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 5762
Jun 20 18:41:31 dist-test-slave-d645 krb5kdc[5389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980888, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.115.3@KRBTEST.COM
I20260620 18:41:31.198465 5905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:31.183193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50408 (local address 127.23.115.3:41761)
0620 18:41:31.183388 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:31.183393 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:31.183415 (+ 22us) server_negotiation.cc:407] Connection header received
0620 18:41:31.183565 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:31.183569 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:31.183636 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:31.183742 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:31.185028 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:31.185723 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:31.187057 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:31.187460 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:31.187559 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:31.192306 (+ 4747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:31.192358 (+ 52us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:31.192393 (+ 35us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:31.192506 (+ 113us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:31.195919 (+ 3413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:31.196645 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:31.196652 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:31.196655 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:31.196809 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:31.197111 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:31.197114 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:31.197116 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:31.197420 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:31.197641 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:31.197952 (+ 311us) server_negotiation.cc:299] Negotiation successful
0620 18:41:31.198139 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 18:41:31.199147 5885 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:41:31.287173 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 5762
I20260620 18:41:31.300403 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 5402
I20260620 18:41:31.301656 5464 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:41:31.415278 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 5402
I20260620 18:41:31.434911 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.115.62:35467
--webserver_interface=127.23.115.62
--webserver_port=45795
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab
--principal=kudu/127.23.115.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:33195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.115.62:35467
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:41:31.764065 5940 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:41:31.764375 5940 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:41:31.764508 5940 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:41:31.772651 5940 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:41:31.772799 5940 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:41:31.772877 5940 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:41:31.772964 5940 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:41:31.773029 5940 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:41:31.783330 5940 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:33195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.23.115.62:35467
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab
--principal=kudu/127.23.115.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.115.62:35467
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.115.62
--webserver_port=45795
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:41:31.784662 5940 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:41:31.785979 5940 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:41:31.808959 5947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:31.809026 5946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:31.809036 5949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:41:31.809638 5940 server_base.cc:1034] running on GCE node
I20260620 18:41:31.810563 5940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:41:31.812258 5940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:41:31.813486 5940 hybrid_clock.cc:648] HybridClock initialized: now 1781980891813436 us; error 33 us; skew 500 ppm
I20260620 18:41:31.813897 5940 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:41:31 dist-test-slave-d645 krb5kdc[5389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980891, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:41:31.817631 5940 init.cc:377] Logged in from keytab as kudu/127.23.115.62@KRBTEST.COM (short username kudu)
I20260620 18:41:31.819517 5940 webserver.cc:460] Webserver started at http://127.23.115.62:45795/ using document root <none> and password file <none>
I20260620 18:41:31.820276 5940 fs_manager.cc:359] Metadata directory not provided
I20260620 18:41:31.820403 5940 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:41:31.826978 5940 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 18:41:31.830292 5956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:31.831903 5940 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 18:41:31.832084 5940 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/master-0/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/master-0/wal
uuid: "a15618092ac84abc82a0f318082d7455"
format_stamp: "Formatted at 2026-06-20 18:41:28 on dist-test-slave-d645"
server_key: "a4bfc05ff5542979236ba117c8b09d67"
server_key_iv: "18565b14a5c71413dce244e0ab893eac"
server_key_version: "encryptionkey@0"
I20260620 18:41:31.833176 5940 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:41:31.858160 5940 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:41:31.862680 5940 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:41:31.863003 5940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:41:31.881362 5940 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.62:35467
I20260620 18:41:31.881403 6007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.62:35467 every 8 connection(s)
I20260620 18:41:31.882835 5940 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
I20260620 18:41:31.888818 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 5940
I20260620 18:41:31.889846 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.115.1:46031
--local_ip_for_outbound_sockets=127.23.115.1
--tserver_master_addrs=127.23.115.62:35467
--webserver_port=43901
--webserver_interface=127.23.115.1
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.1.keytab
--principal=kudu/127.23.115.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:33195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:41:31.890224 6008 sys_catalog.cc:263] Verifying existing consensus state
I20260620 18:41:31.894856 6008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a15618092ac84abc82a0f318082d7455: Bootstrap starting.
I20260620 18:41:31.915925 6008 log.cc:824] T 00000000000000000000000000000000 P a15618092ac84abc82a0f318082d7455: Log is configured to *not* fsync() on all Append() calls
I20260620 18:41:31.929345 6008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a15618092ac84abc82a0f318082d7455: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 18:41:31.929900 6008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a15618092ac84abc82a0f318082d7455: Bootstrap complete.
I20260620 18:41:31.939273 6008 raft_consensus.cc:348] T 00000000000000000000000000000000 P a15618092ac84abc82a0f318082d7455 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a15618092ac84abc82a0f318082d7455" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 35467 } }
I20260620 18:41:31.940112 6008 raft_consensus.cc:729] T 00000000000000000000000000000000 P a15618092ac84abc82a0f318082d7455 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a15618092ac84abc82a0f318082d7455, State: Initialized, Role: FOLLOWER
I20260620 18:41:31.940713 6008 consensus_queue.cc:260] T 00000000000000000000000000000000 P a15618092ac84abc82a0f318082d7455 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a15618092ac84abc82a0f318082d7455" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 35467 } }
I20260620 18:41:31.940932 6008 raft_consensus.cc:388] T 00000000000000000000000000000000 P a15618092ac84abc82a0f318082d7455 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:41:31.941045 6008 raft_consensus.cc:482] T 00000000000000000000000000000000 P a15618092ac84abc82a0f318082d7455 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:41:31.941200 6008 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a15618092ac84abc82a0f318082d7455 [term 1 FOLLOWER]: Advancing to term 2
I20260620 18:41:31.945047 6008 raft_consensus.cc:504] T 00000000000000000000000000000000 P a15618092ac84abc82a0f318082d7455 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a15618092ac84abc82a0f318082d7455" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 35467 } }
I20260620 18:41:31.945560 6008 leader_election.cc:302] T 00000000000000000000000000000000 P a15618092ac84abc82a0f318082d7455 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a15618092ac84abc82a0f318082d7455; no voters:
I20260620 18:41:31.946163 6008 leader_election.cc:288] T 00000000000000000000000000000000 P a15618092ac84abc82a0f318082d7455 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 18:41:31.946308 6012 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a15618092ac84abc82a0f318082d7455 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 18:41:31.947372 6012 raft_consensus.cc:686] T 00000000000000000000000000000000 P a15618092ac84abc82a0f318082d7455 [term 2 LEADER]: Becoming Leader. State: Replica: a15618092ac84abc82a0f318082d7455, State: Running, Role: LEADER
I20260620 18:41:31.947968 6012 consensus_queue.cc:237] T 00000000000000000000000000000000 P a15618092ac84abc82a0f318082d7455 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a15618092ac84abc82a0f318082d7455" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 35467 } }
I20260620 18:41:31.948333 6008 sys_catalog.cc:564] T 00000000000000000000000000000000 P a15618092ac84abc82a0f318082d7455 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:41:31.950496 6014 sys_catalog.cc:455] T 00000000000000000000000000000000 P a15618092ac84abc82a0f318082d7455 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a15618092ac84abc82a0f318082d7455. Latest consensus state: current_term: 2 leader_uuid: "a15618092ac84abc82a0f318082d7455" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a15618092ac84abc82a0f318082d7455" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 35467 } } }
I20260620 18:41:31.950884 6014 sys_catalog.cc:458] T 00000000000000000000000000000000 P a15618092ac84abc82a0f318082d7455 [sys.catalog]: This master's current role is: LEADER
I20260620 18:41:31.951433 6013 sys_catalog.cc:455] T 00000000000000000000000000000000 P a15618092ac84abc82a0f318082d7455 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a15618092ac84abc82a0f318082d7455" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a15618092ac84abc82a0f318082d7455" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 35467 } } }
I20260620 18:41:31.951730 6013 sys_catalog.cc:458] T 00000000000000000000000000000000 P a15618092ac84abc82a0f318082d7455 [sys.catalog]: This master's current role is: LEADER
I20260620 18:41:31.953267 6020 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:41:31.961588 6020 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:41:31.963546 6020 catalog_manager.cc:1259] Loaded cluster ID: 2ce60297219545ebaf2008b08b4ce1ed
I20260620 18:41:31.963723 6020 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:41:31.967273 6020 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260620 18:41:31.986771 6020 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260620 18:41:31.997224 6020 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:41:31.999320 6020 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a15618092ac84abc82a0f318082d7455: Loaded TSK: 0
I20260620 18:41:32.000880 6020 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 18:41:32.237506 6010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:41:32.237823 6010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:41:32.237967 6010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:41:32.246503 6010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:41:32.246637 6010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:41:32.246793 6010 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.115.1
I20260620 18:41:32.257449 6010 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:33195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.1.keytab
--principal=kudu/127.23.115.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.115.1:46031
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.115.1
--webserver_port=43901
--tserver_master_addrs=127.23.115.62:35467
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.115.1
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:41:32.258736 6010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:41:32.260021 6010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:41:32.274363 6035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:32.274331 6036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:32.275511 6038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:41:32.275633 6010 server_base.cc:1034] running on GCE node
I20260620 18:41:32.276271 6010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:41:32.277248 6010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:41:32.278467 6010 hybrid_clock.cc:648] HybridClock initialized: now 1781980892278416 us; error 39 us; skew 500 ppm
I20260620 18:41:32.278792 6010 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:41:32 dist-test-slave-d645 krb5kdc[5389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980892, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:41:32.281831 6010 init.cc:377] Logged in from keytab as kudu/127.23.115.1@KRBTEST.COM (short username kudu)
I20260620 18:41:32.283766 6010 webserver.cc:460] Webserver started at http://127.23.115.1:43901/ using document root <none> and password file <none>
I20260620 18:41:32.284435 6010 fs_manager.cc:359] Metadata directory not provided
I20260620 18:41:32.284572 6010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:41:32.289896 6010 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 18:41:32.292651 6045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:32.294345 6010 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260620 18:41:32.294519 6010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-0/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-0/wal
uuid: "f758242b8e3d4b2e8742c80587dbc12c"
format_stamp: "Formatted at 2026-06-20 18:41:29 on dist-test-slave-d645"
server_key: "e7283e95ed8c97ee2dd7a36dcede1350"
server_key_iv: "38f50c2f96fc434ee8cafffd757f8086"
server_key_version: "encryptionkey@0"
I20260620 18:41:32.295583 6010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:41:32.308372 6010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:41:32.312325 6010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:41:32.312573 6010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:41:32.313562 6010 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:41:32.314920 6010 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:41:32.315061 6010 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:32.315165 6010 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:41:32.315234 6010 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:32.347160 6010 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.1:46031
I20260620 18:41:32.347231 6157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.1:46031 every 8 connection(s)
I20260620 18:41:32.348527 6010 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-0/data/info.pb
I20260620 18:41:32.355249 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 6010
I20260620 18:41:32.355854 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.115.2:36887
--local_ip_for_outbound_sockets=127.23.115.2
--tserver_master_addrs=127.23.115.62:35467
--webserver_port=41981
--webserver_interface=127.23.115.2
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.2.keytab
--principal=kudu/127.23.115.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:33195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 18:41:32 dist-test-slave-d645 krb5kdc[5389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980892, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.1@KRBTEST.COM for kudu/127.23.115.62@KRBTEST.COM
I20260620 18:41:32.371572 6160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:32.351283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.1:56213 (local address 127.23.115.62:35467)
0620 18:41:32.351854 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:32.351871 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:32.352456 (+ 585us) server_negotiation.cc:407] Connection header received
0620 18:41:32.354405 (+ 1949us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:32.354419 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:32.354809 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:32.355174 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:32.356724 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:32.358011 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:32.359888 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:32.360589 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:32.360774 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:32.364312 (+ 3538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:32.364320 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:32.364335 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:32.364377 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:32.367304 (+ 2927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:32.368038 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:32.368044 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:32.368049 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:32.368214 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:32.368616 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:32.368619 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:32.368621 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:32.369056 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:32.369244 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:32.369969 (+ 725us) server_negotiation.cc:299] Negotiation successful
0620 18:41:32.370337 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":221,"threads_started":1}
I20260620 18:41:32.371572 6161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:32.351494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:35467 (local address 127.23.115.1:56213)
0620 18:41:32.352227 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:32.352268 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:32.353904 (+ 1636us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:32.355347 (+ 1443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:32.355358 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:32.355932 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:41:32.356497 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:32.356514 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:32.358188 (+ 1674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:32.358192 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:32.359655 (+ 1463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:32.359673 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:32.360846 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:32.360854 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:32.361095 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:32.361732 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:41:32.361755 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:41:32.364045 (+ 2290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:41:32.367444 (+ 3399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:32.367453 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:32.367468 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:32.367899 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:32.368321 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:32.368327 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:32.368331 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:32.368502 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:32.369197 (+ 695us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:41:32.369204 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:41:32.369502 (+ 298us) client_negotiation.cc:769] Sending connection context
0620 18:41:32.369738 (+ 236us) client_negotiation.cc:240] Negotiation successful
0620 18:41:32.370276 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":300,"threads_started":1}
I20260620 18:41:32.374624 6158 heartbeater.cc:347] Connected to a master server at 127.23.115.62:35467
I20260620 18:41:32.374891 6158 heartbeater.cc:464] Registering TS with master...
I20260620 18:41:32.375612 6158 heartbeater.cc:511] Master 127.23.115.62:35467 requested a full tablet report, sending...
I20260620 18:41:32.377820 5973 ts_manager.cc:194] Registered new tserver with Master: f758242b8e3d4b2e8742c80587dbc12c (127.23.115.1:46031)
I20260620 18:41:32.379935 5973 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.115.1@KRBTEST.COM'} at 127.23.115.1:56213
W20260620 18:41:32.678145 6162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:41:32.678476 6162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:41:32.678632 6162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:41:32.686595 6162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:41:32.686702 6162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:41:32.686842 6162 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.115.2
I20260620 18:41:32.696610 6162 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:33195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.2.keytab
--principal=kudu/127.23.115.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.115.2:36887
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.115.2
--webserver_port=41981
--tserver_master_addrs=127.23.115.62:35467
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.115.2
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:41:32.697897 6162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:41:32.699246 6162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:41:32.713289 6167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:32.713274 6170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:32.713419 6168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:41:32.714250 6162 server_base.cc:1034] running on GCE node
I20260620 18:41:32.714986 6162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:41:32.716096 6162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:41:32.717331 6162 hybrid_clock.cc:648] HybridClock initialized: now 1781980892717248 us; error 76 us; skew 500 ppm
I20260620 18:41:32.717648 6162 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:41:32 dist-test-slave-d645 krb5kdc[5389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980892, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:41:32.720883 6162 init.cc:377] Logged in from keytab as kudu/127.23.115.2@KRBTEST.COM (short username kudu)
I20260620 18:41:32.722397 6162 webserver.cc:460] Webserver started at http://127.23.115.2:41981/ using document root <none> and password file <none>
I20260620 18:41:32.723107 6162 fs_manager.cc:359] Metadata directory not provided
I20260620 18:41:32.723248 6162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:41:32.728989 6162 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 18:41:32.731716 6177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:32.732957 6162 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:41:32.733106 6162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-1/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-1/wal
uuid: "f1e747c088b146c3a4eb96b53a980fb7"
format_stamp: "Formatted at 2026-06-20 18:41:29 on dist-test-slave-d645"
server_key: "c0a8216d3a3a482d534a45555a5a35b1"
server_key_iv: "bb1d71dc245e2bfd52b199f6f4c7a19d"
server_key_version: "encryptionkey@0"
I20260620 18:41:32.734045 6162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:41:32.769411 6162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:41:32.773527 6162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:41:32.773869 6162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:41:32.775291 6162 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:41:32.776845 6162 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:41:32.776968 6162 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:32.777077 6162 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:41:32.777137 6162 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:32.809101 6162 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.2:36887
I20260620 18:41:32.809187 6289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.2:36887 every 8 connection(s)
I20260620 18:41:32.810448 6162 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-1/data/info.pb
I20260620 18:41:32.818132 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 6162
I20260620 18:41:32.818840 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.115.3:41761
--local_ip_for_outbound_sockets=127.23.115.3
--tserver_master_addrs=127.23.115.62:35467
--webserver_port=43223
--webserver_interface=127.23.115.3
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.3.keytab
--principal=kudu/127.23.115.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:33195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 18:41:32 dist-test-slave-d645 krb5kdc[5389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980892, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.2@KRBTEST.COM for kudu/127.23.115.62@KRBTEST.COM
I20260620 18:41:32.831724 6160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:32.812900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.2:60165 (local address 127.23.115.62:35467)
0620 18:41:32.813094 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:32.813100 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:32.814270 (+ 1170us) server_negotiation.cc:407] Connection header received
0620 18:41:32.816043 (+ 1773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:32.816047 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:32.816117 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:32.816243 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:32.817585 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:32.818627 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:32.820367 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:32.820964 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:32.821099 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:32.825269 (+ 4170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:32.825277 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:32.825282 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:32.825328 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:32.828089 (+ 2761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:32.829199 (+ 1110us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:32.829204 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:32.829206 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:32.829337 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:32.829874 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:32.829878 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:32.829881 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:32.830179 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:32.830300 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:32.831289 (+ 989us) server_negotiation.cc:299] Negotiation successful
0620 18:41:32.831469 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 18:41:32.832962 6292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:32.813414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:35467 (local address 127.23.115.2:60165)
0620 18:41:32.814078 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:32.814118 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:32.815777 (+ 1659us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:32.816395 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:32.816402 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:32.816897 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:41:32.817413 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:32.817431 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:32.818825 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:32.818829 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:32.820196 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:32.820219 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:32.821189 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:32.821194 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:32.821430 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:32.822203 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:41:32.822231 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:41:32.824944 (+ 2713us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:41:32.828272 (+ 3328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:32.828283 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:32.828302 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:32.829011 (+ 709us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:32.829443 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:32.829448 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:32.829452 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:32.829694 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:32.830303 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:41:32.830311 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:41:32.830766 (+ 455us) client_negotiation.cc:769] Sending connection context
0620 18:41:32.831091 (+ 325us) client_negotiation.cc:240] Negotiation successful
0620 18:41:32.831647 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":255,"threads_started":1}
I20260620 18:41:32.834503 6290 heartbeater.cc:347] Connected to a master server at 127.23.115.62:35467
I20260620 18:41:32.834820 6290 heartbeater.cc:464] Registering TS with master...
I20260620 18:41:32.835593 6290 heartbeater.cc:511] Master 127.23.115.62:35467 requested a full tablet report, sending...
I20260620 18:41:32.837327 5973 ts_manager.cc:194] Registered new tserver with Master: f1e747c088b146c3a4eb96b53a980fb7 (127.23.115.2:36887)
I20260620 18:41:32.838522 5973 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.115.2@KRBTEST.COM'} at 127.23.115.2:60165
W20260620 18:41:33.141487 6293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:41:33.141762 6293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:41:33.141888 6293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:41:33.149183 6293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:41:33.149293 6293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:41:33.149435 6293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.115.3
I20260620 18:41:33.159260 6293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:33195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.3.keytab
--principal=kudu/127.23.115.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.115.3:41761
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.115.3
--webserver_port=43223
--tserver_master_addrs=127.23.115.62:35467
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.115.3
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:41:33.160533 6293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:41:33.161844 6293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:41:33.177170 6301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:33.177213 6299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:33.177273 6298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:41:33.178476 6293 server_base.cc:1034] running on GCE node
I20260620 18:41:33.179214 6293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:41:33.180186 6293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:41:33.181413 6293 hybrid_clock.cc:648] HybridClock initialized: now 1781980893181352 us; error 55 us; skew 500 ppm
I20260620 18:41:33.181782 6293 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:41:33 dist-test-slave-d645 krb5kdc[5389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980893, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:41:33.185491 6293 init.cc:377] Logged in from keytab as kudu/127.23.115.3@KRBTEST.COM (short username kudu)
I20260620 18:41:33.187168 6293 webserver.cc:460] Webserver started at http://127.23.115.3:43223/ using document root <none> and password file <none>
I20260620 18:41:33.187871 6293 fs_manager.cc:359] Metadata directory not provided
I20260620 18:41:33.188007 6293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:41:33.194499 6293 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.003s
I20260620 18:41:33.198485 6308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:33.200536 6293 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 18:41:33.200726 6293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-2/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-2/wal
uuid: "b32dcc4d34fc44908dd3d879251d0ef0"
format_stamp: "Formatted at 2026-06-20 18:41:30 on dist-test-slave-d645"
server_key: "3adba41137f35f38872966a1dc3cd1e6"
server_key_iv: "6b0c1a0c31be04487a2a6d029ab1eea9"
server_key_version: "encryptionkey@0"
I20260620 18:41:33.201869 6293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:41:33.229111 6293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:41:33.233400 6293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:41:33.233664 6293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:41:33.234726 6293 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:41:33.236110 6293 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:41:33.236289 6293 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:33.236409 6293 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:41:33.236495 6293 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:33.270198 6293 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.3:41761
I20260620 18:41:33.270272 6420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.3:41761 every 8 connection(s)
I20260620 18:41:33.271615 6293 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-2/data/info.pb
I20260620 18:41:33.274472 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 6293
Jun 20 18:41:33 dist-test-slave-d645 krb5kdc[5389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980893, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.3@KRBTEST.COM for kudu/127.23.115.62@KRBTEST.COM
I20260620 18:41:33.291967 6424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:33.275359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41342 (local address 127.23.115.62:35467)
0620 18:41:33.276575 (+ 1216us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:33.276582 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:33.276597 (+ 15us) server_negotiation.cc:407] Connection header received
0620 18:41:33.276658 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:33.276663 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:33.276730 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:33.276822 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:33.278490 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:33.279341 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:33.281200 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:33.281853 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:33.282025 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:33.285471 (+ 3446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:33.285477 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:33.285482 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:33.285519 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:33.288955 (+ 3436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:33.289906 (+ 951us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:33.289911 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:33.289913 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:33.290064 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:33.290478 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:33.290483 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:33.290485 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:33.290783 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:33.291076 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:33.291391 (+ 315us) server_negotiation.cc:299] Negotiation successful
0620 18:41:33.291612 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":1117,"thread_start_us":996,"threads_started":1}
I20260620 18:41:33.294428 6160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:33.274245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.3:42549 (local address 127.23.115.62:35467)
0620 18:41:33.274470 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:33.274476 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:33.275671 (+ 1195us) server_negotiation.cc:407] Connection header received
0620 18:41:33.277792 (+ 2121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:33.277798 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:33.277875 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:33.277997 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:33.280147 (+ 2150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:33.281307 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:33.283480 (+ 2173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:33.284103 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:33.284234 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:33.288562 (+ 4328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:33.288567 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:33.288570 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:33.288614 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:33.291298 (+ 2684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:33.292035 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:33.292041 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:33.292043 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:33.292179 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:33.292670 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:33.292675 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:33.292678 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:33.292933 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:33.293057 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:33.293900 (+ 843us) server_negotiation.cc:299] Negotiation successful
0620 18:41:33.294105 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":73}
I20260620 18:41:33.295459 6423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:33.274723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:35467 (local address 127.23.115.3:42549)
0620 18:41:33.275496 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:33.275555 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:33.277533 (+ 1978us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:33.278529 (+ 996us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:33.278542 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:33.279375 (+ 833us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:41:33.279945 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:33.279965 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:33.281514 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:33.281520 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:33.283256 (+ 1736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:33.283271 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:33.284365 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:33.284371 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:33.284577 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:33.285267 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:41:33.285291 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:41:33.288289 (+ 2998us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:41:33.291435 (+ 3146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:33.291445 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:33.291463 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:33.291933 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:33.292299 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:33.292303 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:33.292305 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:33.292542 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:33.293068 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:41:33.293077 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:41:33.293541 (+ 464us) client_negotiation.cc:769] Sending connection context
0620 18:41:33.293780 (+ 239us) client_negotiation.cc:240] Negotiation successful
0620 18:41:33.294320 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":297,"threads_started":1}
I20260620 18:41:33.297251 6421 heartbeater.cc:347] Connected to a master server at 127.23.115.62:35467
I20260620 18:41:33.297580 6421 heartbeater.cc:464] Registering TS with master...
I20260620 18:41:33.298734 6421 heartbeater.cc:511] Master 127.23.115.62:35467 requested a full tablet report, sending...
I20260620 18:41:33.300356 5973 ts_manager.cc:194] Registered new tserver with Master: b32dcc4d34fc44908dd3d879251d0ef0 (127.23.115.3:41761)
I20260620 18:41:33.301177 5973 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.115.3@KRBTEST.COM'} at 127.23.115.3:42549
I20260620 18:41:33.307029 24012 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:41:33.314558 6160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:33.307912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41348 (local address 127.23.115.62:35467)
0620 18:41:33.308101 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:33.308108 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:33.308529 (+ 421us) server_negotiation.cc:407] Connection header received
0620 18:41:33.308808 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:33.308813 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:33.308955 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:33.309126 (+ 171us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:41:33.309650 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:33.310651 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:33.312281 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:33.312786 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:33.312896 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:33.313441 (+ 545us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:41:33.313935 (+ 494us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:41:33.314046 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:33.314224 (+ 178us) server_negotiation.cc:299] Negotiation successful
0620 18:41:33.314297 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 18:41:33.315534 5973 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:41348:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:41:33.322820 5973 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:41:33.350013 6434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:33.335890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34684 (local address 127.23.115.2:36887)
0620 18:41:33.339743 (+ 3853us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:33.339753 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:33.339782 (+ 29us) server_negotiation.cc:407] Connection header received
0620 18:41:33.339895 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:33.339903 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:33.340183 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:33.341182 (+ 999us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:41:33.341408 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:33.343062 (+ 1654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:33.346538 (+ 3476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:33.347665 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:33.347790 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:33.347980 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:33.349521 (+ 1541us) server_negotiation.cc:299] Negotiation successful
0620 18:41:33.349695 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3688,"thread_start_us":3543,"threads_started":1}
I20260620 18:41:33.351125 6435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:33.335357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.2:36887 (local address 127.0.0.1:34684)
0620 18:41:33.336848 (+ 1491us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:33.336866 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:33.336971 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:33.340441 (+ 3470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:33.340450 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:33.340483 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:41:33.340940 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:33.340953 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:33.343250 (+ 2297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:33.343257 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:33.346265 (+ 3008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:33.346301 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:33.348986 (+ 2685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:33.348996 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:33.349226 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:33.349279 (+ 53us) client_negotiation.cc:769] Sending connection context
0620 18:41:33.350406 (+ 1127us) client_negotiation.cc:240] Negotiation successful
0620 18:41:33.350637 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1366,"mutex_wait_us":3,"thread_start_us":8022,"threads_started":1}
I20260620 18:41:33.351949 6433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:33.336172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.1:46031 (local address 127.0.0.1:33350)
0620 18:41:33.343658 (+ 7486us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:33.343675 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:33.343785 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:33.344759 (+ 974us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:33.344763 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:33.344793 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:41:33.345129 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:33.345137 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:33.347012 (+ 1875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:33.347018 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:33.349145 (+ 2127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:33.349160 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:33.351092 (+ 1932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:33.351100 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:33.351311 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:33.351348 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 18:41:33.351426 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 18:41:33.351565 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7367,"mutex_wait_us":56,"thread_start_us":693,"threads_started":1}
I20260620 18:41:33.353041 6436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:33.336689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33350 (local address 127.23.115.1:46031)
0620 18:41:33.339380 (+ 2691us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:33.339390 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:33.343922 (+ 4532us) server_negotiation.cc:407] Connection header received
0620 18:41:33.344046 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:33.344054 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:33.344301 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:33.344491 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:41:33.345278 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:33.346842 (+ 1564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:33.349336 (+ 2494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:33.350718 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:33.352247 (+ 1529us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:33.352470 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:33.352542 (+ 72us) server_negotiation.cc:299] Negotiation successful
0620 18:41:33.352696 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2511,"thread_start_us":2394,"threads_started":1}
I20260620 18:41:33.354660 6431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:33.334844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.3:41761 (local address 127.0.0.1:50424)
0620 18:41:33.335742 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:33.335783 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:33.335958 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:33.348081 (+ 12123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:33.348085 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:33.348112 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:41:33.348545 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:33.348558 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:33.350452 (+ 1894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:33.350457 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:33.352340 (+ 1883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:33.352352 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:33.353906 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:33.353914 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:33.354096 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:33.354128 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 18:41:33.354193 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 18:41:33.354324 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":628,"mutex_wait_us":6,"thread_start_us":460,"threads_started":1}
I20260620 18:41:33.355754 6432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:33.335164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50424 (local address 127.23.115.3:41761)
0620 18:41:33.345738 (+ 10574us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:33.345746 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:33.345779 (+ 33us) server_negotiation.cc:407] Connection header received
0620 18:41:33.345862 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:33.345869 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:33.346081 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:33.346261 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:41:33.348787 (+ 2526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:33.350301 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:33.352500 (+ 2199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:33.353668 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:33.354974 (+ 1306us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:33.355193 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:33.355257 (+ 64us) server_negotiation.cc:299] Negotiation successful
0620 18:41:33.355413 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10395,"thread_start_us":10222,"threads_started":1}
I20260620 18:41:33.356678 6225 tablet_service.cc:1467] Processing CreateTablet for tablet b32fa4a4ac5c45749a7f2f0859508c8a (DEFAULT_TABLE table=test-table [id=f634643bd89d484c948cd7a9b1051d9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:41:33.356678 6093 tablet_service.cc:1467] Processing CreateTablet for tablet b32fa4a4ac5c45749a7f2f0859508c8a (DEFAULT_TABLE table=test-table [id=f634643bd89d484c948cd7a9b1051d9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:41:33.358062 6225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b32fa4a4ac5c45749a7f2f0859508c8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:41:33.358081 6093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b32fa4a4ac5c45749a7f2f0859508c8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:41:33.358413 6356 tablet_service.cc:1467] Processing CreateTablet for tablet b32fa4a4ac5c45749a7f2f0859508c8a (DEFAULT_TABLE table=test-table [id=f634643bd89d484c948cd7a9b1051d9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:41:33.359611 6356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b32fa4a4ac5c45749a7f2f0859508c8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:41:33.371315 6437 tablet_bootstrap.cc:492] T b32fa4a4ac5c45749a7f2f0859508c8a P b32dcc4d34fc44908dd3d879251d0ef0: Bootstrap starting.
I20260620 18:41:33.374421 6437 tablet_bootstrap.cc:654] T b32fa4a4ac5c45749a7f2f0859508c8a P b32dcc4d34fc44908dd3d879251d0ef0: Neither blocks nor log segments found. Creating new log.
I20260620 18:41:33.375242 6438 tablet_bootstrap.cc:492] T b32fa4a4ac5c45749a7f2f0859508c8a P f1e747c088b146c3a4eb96b53a980fb7: Bootstrap starting.
I20260620 18:41:33.376051 6439 tablet_bootstrap.cc:492] T b32fa4a4ac5c45749a7f2f0859508c8a P f758242b8e3d4b2e8742c80587dbc12c: Bootstrap starting.
I20260620 18:41:33.377046 6437 log.cc:824] T b32fa4a4ac5c45749a7f2f0859508c8a P b32dcc4d34fc44908dd3d879251d0ef0: Log is configured to *not* fsync() on all Append() calls
I20260620 18:41:33.379146 6438 tablet_bootstrap.cc:654] T b32fa4a4ac5c45749a7f2f0859508c8a P f1e747c088b146c3a4eb96b53a980fb7: Neither blocks nor log segments found. Creating new log.
I20260620 18:41:33.379253 6439 tablet_bootstrap.cc:654] T b32fa4a4ac5c45749a7f2f0859508c8a P f758242b8e3d4b2e8742c80587dbc12c: Neither blocks nor log segments found. Creating new log.
I20260620 18:41:33.380138 6438 log.cc:824] T b32fa4a4ac5c45749a7f2f0859508c8a P f1e747c088b146c3a4eb96b53a980fb7: Log is configured to *not* fsync() on all Append() calls
I20260620 18:41:33.380328 6437 tablet_bootstrap.cc:492] T b32fa4a4ac5c45749a7f2f0859508c8a P b32dcc4d34fc44908dd3d879251d0ef0: No bootstrap required, opened a new log
I20260620 18:41:33.380414 6439 log.cc:824] T b32fa4a4ac5c45749a7f2f0859508c8a P f758242b8e3d4b2e8742c80587dbc12c: Log is configured to *not* fsync() on all Append() calls
I20260620 18:41:33.380605 6437 ts_tablet_manager.cc:1397] T b32fa4a4ac5c45749a7f2f0859508c8a P b32dcc4d34fc44908dd3d879251d0ef0: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260620 18:41:33.382851 6438 tablet_bootstrap.cc:492] T b32fa4a4ac5c45749a7f2f0859508c8a P f1e747c088b146c3a4eb96b53a980fb7: No bootstrap required, opened a new log
I20260620 18:41:33.383065 6158 heartbeater.cc:503] Master 127.23.115.62:35467 was elected leader, sending a full tablet report...
I20260620 18:41:33.383145 6438 ts_tablet_manager.cc:1397] T b32fa4a4ac5c45749a7f2f0859508c8a P f1e747c088b146c3a4eb96b53a980fb7: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.003s
I20260620 18:41:33.384969 6439 tablet_bootstrap.cc:492] T b32fa4a4ac5c45749a7f2f0859508c8a P f758242b8e3d4b2e8742c80587dbc12c: No bootstrap required, opened a new log
I20260620 18:41:33.385277 6439 ts_tablet_manager.cc:1397] T b32fa4a4ac5c45749a7f2f0859508c8a P f758242b8e3d4b2e8742c80587dbc12c: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.006s
I20260620 18:41:33.391268 6438 raft_consensus.cc:348] T b32fa4a4ac5c45749a7f2f0859508c8a P f1e747c088b146c3a4eb96b53a980fb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b32dcc4d34fc44908dd3d879251d0ef0" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 41761 } } peers { permanent_uuid: "f758242b8e3d4b2e8742c80587dbc12c" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 46031 } } peers { permanent_uuid: "f1e747c088b146c3a4eb96b53a980fb7" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 36887 } }
I20260620 18:41:33.391629 6438 raft_consensus.cc:374] T b32fa4a4ac5c45749a7f2f0859508c8a P f1e747c088b146c3a4eb96b53a980fb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:41:33.391423 6437 raft_consensus.cc:348] T b32fa4a4ac5c45749a7f2f0859508c8a P b32dcc4d34fc44908dd3d879251d0ef0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b32dcc4d34fc44908dd3d879251d0ef0" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 41761 } } peers { permanent_uuid: "f758242b8e3d4b2e8742c80587dbc12c" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 46031 } } peers { permanent_uuid: "f1e747c088b146c3a4eb96b53a980fb7" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 36887 } }
I20260620 18:41:33.391729 6438 raft_consensus.cc:729] T b32fa4a4ac5c45749a7f2f0859508c8a P f1e747c088b146c3a4eb96b53a980fb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e747c088b146c3a4eb96b53a980fb7, State: Initialized, Role: FOLLOWER
I20260620 18:41:33.391839 6437 raft_consensus.cc:374] T b32fa4a4ac5c45749a7f2f0859508c8a P b32dcc4d34fc44908dd3d879251d0ef0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:41:33.391932 6437 raft_consensus.cc:729] T b32fa4a4ac5c45749a7f2f0859508c8a P b32dcc4d34fc44908dd3d879251d0ef0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b32dcc4d34fc44908dd3d879251d0ef0, State: Initialized, Role: FOLLOWER
I20260620 18:41:33.392336 6438 consensus_queue.cc:260] T b32fa4a4ac5c45749a7f2f0859508c8a P f1e747c088b146c3a4eb96b53a980fb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b32dcc4d34fc44908dd3d879251d0ef0" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 41761 } } peers { permanent_uuid: "f758242b8e3d4b2e8742c80587dbc12c" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 46031 } } peers { permanent_uuid: "f1e747c088b146c3a4eb96b53a980fb7" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 36887 } }
I20260620 18:41:33.392498 6437 consensus_queue.cc:260] T b32fa4a4ac5c45749a7f2f0859508c8a P b32dcc4d34fc44908dd3d879251d0ef0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b32dcc4d34fc44908dd3d879251d0ef0" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 41761 } } peers { permanent_uuid: "f758242b8e3d4b2e8742c80587dbc12c" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 46031 } } peers { permanent_uuid: "f1e747c088b146c3a4eb96b53a980fb7" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 36887 } }
I20260620 18:41:33.393679 6421 heartbeater.cc:503] Master 127.23.115.62:35467 was elected leader, sending a full tablet report...
I20260620 18:41:33.394256 6437 ts_tablet_manager.cc:1428] T b32fa4a4ac5c45749a7f2f0859508c8a P b32dcc4d34fc44908dd3d879251d0ef0: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260620 18:41:33.395445 6290 heartbeater.cc:503] Master 127.23.115.62:35467 was elected leader, sending a full tablet report...
I20260620 18:41:33.395349 6439 raft_consensus.cc:348] T b32fa4a4ac5c45749a7f2f0859508c8a P f758242b8e3d4b2e8742c80587dbc12c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b32dcc4d34fc44908dd3d879251d0ef0" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 41761 } } peers { permanent_uuid: "f758242b8e3d4b2e8742c80587dbc12c" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 46031 } } peers { permanent_uuid: "f1e747c088b146c3a4eb96b53a980fb7" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 36887 } }
I20260620 18:41:33.395762 6438 ts_tablet_manager.cc:1428] T b32fa4a4ac5c45749a7f2f0859508c8a P f1e747c088b146c3a4eb96b53a980fb7: Time spent starting tablet: real 0.013s user 0.011s sys 0.001s
I20260620 18:41:33.395802 6439 raft_consensus.cc:374] T b32fa4a4ac5c45749a7f2f0859508c8a P f758242b8e3d4b2e8742c80587dbc12c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:41:33.395893 6439 raft_consensus.cc:729] T b32fa4a4ac5c45749a7f2f0859508c8a P f758242b8e3d4b2e8742c80587dbc12c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f758242b8e3d4b2e8742c80587dbc12c, State: Initialized, Role: FOLLOWER
I20260620 18:41:33.396420 6439 consensus_queue.cc:260] T b32fa4a4ac5c45749a7f2f0859508c8a P f758242b8e3d4b2e8742c80587dbc12c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b32dcc4d34fc44908dd3d879251d0ef0" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 41761 } } peers { permanent_uuid: "f758242b8e3d4b2e8742c80587dbc12c" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 46031 } } peers { permanent_uuid: "f1e747c088b146c3a4eb96b53a980fb7" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 36887 } }
I20260620 18:41:33.398116 6439 ts_tablet_manager.cc:1428] T b32fa4a4ac5c45749a7f2f0859508c8a P f758242b8e3d4b2e8742c80587dbc12c: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260620 18:41:33.413527 6445 raft_consensus.cc:482] T b32fa4a4ac5c45749a7f2f0859508c8a P f758242b8e3d4b2e8742c80587dbc12c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:41:33.413827 6445 raft_consensus.cc:504] T b32fa4a4ac5c45749a7f2f0859508c8a P f758242b8e3d4b2e8742c80587dbc12c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b32dcc4d34fc44908dd3d879251d0ef0" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 41761 } } peers { permanent_uuid: "f758242b8e3d4b2e8742c80587dbc12c" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 46031 } } peers { permanent_uuid: "f1e747c088b146c3a4eb96b53a980fb7" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 36887 } }
I20260620 18:41:33.416538 6445 leader_election.cc:288] T b32fa4a4ac5c45749a7f2f0859508c8a P f758242b8e3d4b2e8742c80587dbc12c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b32dcc4d34fc44908dd3d879251d0ef0 (127.23.115.3:41761), f1e747c088b146c3a4eb96b53a980fb7 (127.23.115.2:36887)
I20260620 18:41:33.425181 6432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:33.416149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.1:56721 (local address 127.23.115.3:41761)
0620 18:41:33.416353 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:33.416359 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:33.417165 (+ 806us) server_negotiation.cc:407] Connection header received
0620 18:41:33.417321 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:33.417326 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:33.417404 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:33.417506 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:41:33.418660 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:33.419784 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:33.422745 (+ 2961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:33.424012 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:33.424176 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:33.424254 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:33.424781 (+ 527us) server_negotiation.cc:299] Negotiation successful
0620 18:41:33.424932 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 18:41:33.425181 6434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:33.416042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.1:57741 (local address 127.23.115.2:36887)
0620 18:41:33.416241 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:33.416248 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:33.417174 (+ 926us) server_negotiation.cc:407] Connection header received
0620 18:41:33.417345 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:33.417349 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:33.417419 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:33.417506 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:41:33.418672 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:33.419784 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:33.422741 (+ 2957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:33.424012 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:33.424184 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:33.424254 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:33.424783 (+ 529us) server_negotiation.cc:299] Negotiation successful
0620 18:41:33.424921 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 18:41:33.425371 6447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:33.415867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.3:41761 (local address 127.23.115.1:56721)
0620 18:41:33.417004 (+ 1137us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:33.417025 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:33.417188 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:33.417686 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:33.417694 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:33.417812 (+ 118us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:41:33.418477 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:33.418488 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:33.419927 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:33.419933 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:33.422563 (+ 2630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:33.422572 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:33.424275 (+ 1703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:33.424282 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:33.424531 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:33.424595 (+ 64us) client_negotiation.cc:769] Sending connection context
0620 18:41:33.424698 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 18:41:33.424871 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":992,"mutex_wait_us":203,"spinlock_wait_cycles":40192,"thread_start_us":819,"threads_started":1}
I20260620 18:41:33.425617 6446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:33.415992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.2:36887 (local address 127.23.115.1:57741)
0620 18:41:33.417064 (+ 1072us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:33.417078 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:33.417230 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:33.417686 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:33.417694 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:33.417789 (+ 95us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:41:33.418437 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:33.418454 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:33.419929 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:33.419934 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:33.422553 (+ 2619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:33.422572 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:33.424291 (+ 1719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:33.424296 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:33.424481 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:33.424594 (+ 113us) client_negotiation.cc:769] Sending connection context
0620 18:41:33.424701 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 18:41:33.424871 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":976,"mutex_wait_us":287,"spinlock_wait_cycles":95744,"thread_start_us":395,"threads_started":1}
I20260620 18:41:33.426301 6376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b32fa4a4ac5c45749a7f2f0859508c8a" candidate_uuid: "f758242b8e3d4b2e8742c80587dbc12c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b32dcc4d34fc44908dd3d879251d0ef0" is_pre_election: true
I20260620 18:41:33.426514 6245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b32fa4a4ac5c45749a7f2f0859508c8a" candidate_uuid: "f758242b8e3d4b2e8742c80587dbc12c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e747c088b146c3a4eb96b53a980fb7" is_pre_election: true
I20260620 18:41:33.426889 6376 raft_consensus.cc:2454] T b32fa4a4ac5c45749a7f2f0859508c8a P b32dcc4d34fc44908dd3d879251d0ef0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f758242b8e3d4b2e8742c80587dbc12c in term 0.
I20260620 18:41:33.426973 6245 raft_consensus.cc:2454] T b32fa4a4ac5c45749a7f2f0859508c8a P f1e747c088b146c3a4eb96b53a980fb7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f758242b8e3d4b2e8742c80587dbc12c in term 0.
I20260620 18:41:33.427598 6049 leader_election.cc:302] T b32fa4a4ac5c45749a7f2f0859508c8a P f758242b8e3d4b2e8742c80587dbc12c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: b32dcc4d34fc44908dd3d879251d0ef0, f1e747c088b146c3a4eb96b53a980fb7, f758242b8e3d4b2e8742c80587dbc12c; no voters:
I20260620 18:41:33.428016 6445 raft_consensus.cc:2781] T b32fa4a4ac5c45749a7f2f0859508c8a P f758242b8e3d4b2e8742c80587dbc12c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:41:33.428167 6445 raft_consensus.cc:482] T b32fa4a4ac5c45749a7f2f0859508c8a P f758242b8e3d4b2e8742c80587dbc12c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:41:33.428278 6445 raft_consensus.cc:3037] T b32fa4a4ac5c45749a7f2f0859508c8a P f758242b8e3d4b2e8742c80587dbc12c [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:41:33.430382 6445 raft_consensus.cc:504] T b32fa4a4ac5c45749a7f2f0859508c8a P f758242b8e3d4b2e8742c80587dbc12c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b32dcc4d34fc44908dd3d879251d0ef0" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 41761 } } peers { permanent_uuid: "f758242b8e3d4b2e8742c80587dbc12c" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 46031 } } peers { permanent_uuid: "f1e747c088b146c3a4eb96b53a980fb7" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 36887 } }
I20260620 18:41:33.431020 6445 leader_election.cc:288] T b32fa4a4ac5c45749a7f2f0859508c8a P f758242b8e3d4b2e8742c80587dbc12c [CANDIDATE]: Term 1 election: Requested vote from peers b32dcc4d34fc44908dd3d879251d0ef0 (127.23.115.3:41761), f1e747c088b146c3a4eb96b53a980fb7 (127.23.115.2:36887)
I20260620 18:41:33.431470 6376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b32fa4a4ac5c45749a7f2f0859508c8a" candidate_uuid: "f758242b8e3d4b2e8742c80587dbc12c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b32dcc4d34fc44908dd3d879251d0ef0"
I20260620 18:41:33.431541 6245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b32fa4a4ac5c45749a7f2f0859508c8a" candidate_uuid: "f758242b8e3d4b2e8742c80587dbc12c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e747c088b146c3a4eb96b53a980fb7"
I20260620 18:41:33.431686 6376 raft_consensus.cc:3037] T b32fa4a4ac5c45749a7f2f0859508c8a P b32dcc4d34fc44908dd3d879251d0ef0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:41:33.431720 6245 raft_consensus.cc:3037] T b32fa4a4ac5c45749a7f2f0859508c8a P f1e747c088b146c3a4eb96b53a980fb7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:41:33.434823 6245 raft_consensus.cc:2454] T b32fa4a4ac5c45749a7f2f0859508c8a P f1e747c088b146c3a4eb96b53a980fb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f758242b8e3d4b2e8742c80587dbc12c in term 1.
I20260620 18:41:33.434823 6376 raft_consensus.cc:2454] T b32fa4a4ac5c45749a7f2f0859508c8a P b32dcc4d34fc44908dd3d879251d0ef0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f758242b8e3d4b2e8742c80587dbc12c in term 1.
I20260620 18:41:33.435295 6047 leader_election.cc:302] T b32fa4a4ac5c45749a7f2f0859508c8a P f758242b8e3d4b2e8742c80587dbc12c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f1e747c088b146c3a4eb96b53a980fb7, f758242b8e3d4b2e8742c80587dbc12c; no voters:
I20260620 18:41:33.435649 6445 raft_consensus.cc:2781] T b32fa4a4ac5c45749a7f2f0859508c8a P f758242b8e3d4b2e8742c80587dbc12c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:41:33.436079 6445 raft_consensus.cc:686] T b32fa4a4ac5c45749a7f2f0859508c8a P f758242b8e3d4b2e8742c80587dbc12c [term 1 LEADER]: Becoming Leader. State: Replica: f758242b8e3d4b2e8742c80587dbc12c, State: Running, Role: LEADER
I20260620 18:41:33.436519 6445 consensus_queue.cc:237] T b32fa4a4ac5c45749a7f2f0859508c8a P f758242b8e3d4b2e8742c80587dbc12c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b32dcc4d34fc44908dd3d879251d0ef0" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 41761 } } peers { permanent_uuid: "f758242b8e3d4b2e8742c80587dbc12c" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 46031 } } peers { permanent_uuid: "f1e747c088b146c3a4eb96b53a980fb7" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 36887 } }
I20260620 18:41:33.441131 5973 catalog_manager.cc:5495] T b32fa4a4ac5c45749a7f2f0859508c8a P f758242b8e3d4b2e8742c80587dbc12c reported cstate change: term changed from 0 to 1, leader changed from <none> to f758242b8e3d4b2e8742c80587dbc12c (127.23.115.1). New cstate: current_term: 1 leader_uuid: "f758242b8e3d4b2e8742c80587dbc12c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b32dcc4d34fc44908dd3d879251d0ef0" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 41761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f758242b8e3d4b2e8742c80587dbc12c" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 46031 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1e747c088b146c3a4eb96b53a980fb7" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 36887 } health_report { overall_health: UNKNOWN } } }
I20260620 18:41:33.473106 6436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:33.467884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33358 (local address 127.23.115.1:46031)
0620 18:41:33.468055 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:33.468060 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:33.468076 (+ 16us) server_negotiation.cc:407] Connection header received
0620 18:41:33.468163 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:33.468166 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:33.468251 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:33.468368 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:41:33.468738 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:33.469532 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:33.471046 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:33.471449 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:33.471558 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:33.472024 (+ 466us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:41:33.472521 (+ 497us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:41:33.472649 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:33.472771 (+ 122us) server_negotiation.cc:299] Negotiation successful
0620 18:41:33.472853 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 18:41:33.482394 6376 raft_consensus.cc:1261] T b32fa4a4ac5c45749a7f2f0859508c8a P b32dcc4d34fc44908dd3d879251d0ef0 [term 1 FOLLOWER]: Refusing update from remote peer f758242b8e3d4b2e8742c80587dbc12c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:41:33.482395 6245 raft_consensus.cc:1261] T b32fa4a4ac5c45749a7f2f0859508c8a P f1e747c088b146c3a4eb96b53a980fb7 [term 1 FOLLOWER]: Refusing update from remote peer f758242b8e3d4b2e8742c80587dbc12c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:41:33.483388 6445 consensus_queue.cc:1035] T b32fa4a4ac5c45749a7f2f0859508c8a P f758242b8e3d4b2e8742c80587dbc12c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b32dcc4d34fc44908dd3d879251d0ef0" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 41761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:41:33.483680 6448 consensus_queue.cc:1035] T b32fa4a4ac5c45749a7f2f0859508c8a P f758242b8e3d4b2e8742c80587dbc12c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1e747c088b146c3a4eb96b53a980fb7" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 36887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:41:33.493978 6451 mvcc.cc:205] Tried to move back new op lower bound from 7298993739693043712 to 7298993739519848448. Current Snapshot: MvccSnapshot[applied={T|T < 7298993739693043712}]
I20260620 18:41:33.507627 5972 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:41348:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:41:33.507901 5972 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:41348:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:41:33.511584 5972 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a15618092ac84abc82a0f318082d7455: Sending DeleteTablet for 3 replicas of tablet b32fa4a4ac5c45749a7f2f0859508c8a
I20260620 18:41:33.512799 6093 tablet_service.cc:1514] Processing DeleteTablet for tablet b32fa4a4ac5c45749a7f2f0859508c8a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:41:33 UTC) from {username='kudu', principal='kudu/127.23.115.62@KRBTEST.COM'} at 127.0.0.1:33350
I20260620 18:41:33.512863 6225 tablet_service.cc:1514] Processing DeleteTablet for tablet b32fa4a4ac5c45749a7f2f0859508c8a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:41:33 UTC) from {username='kudu', principal='kudu/127.23.115.62@KRBTEST.COM'} at 127.0.0.1:34684
I20260620 18:41:33.513018 6356 tablet_service.cc:1514] Processing DeleteTablet for tablet b32fa4a4ac5c45749a7f2f0859508c8a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:41:33 UTC) from {username='kudu', principal='kudu/127.23.115.62@KRBTEST.COM'} at 127.0.0.1:50424
I20260620 18:41:33.514287 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 6010
I20260620 18:41:33.522732 6463 tablet_replica.cc:330] T b32fa4a4ac5c45749a7f2f0859508c8a P b32dcc4d34fc44908dd3d879251d0ef0: stopping tablet replica
I20260620 18:41:33.523005 6462 tablet_replica.cc:330] T b32fa4a4ac5c45749a7f2f0859508c8a P f758242b8e3d4b2e8742c80587dbc12c: stopping tablet replica
I20260620 18:41:33.523465 6462 raft_consensus.cc:2229] T b32fa4a4ac5c45749a7f2f0859508c8a P f758242b8e3d4b2e8742c80587dbc12c [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:41:33.524005 6462 raft_consensus.cc:2258] T b32fa4a4ac5c45749a7f2f0859508c8a P f758242b8e3d4b2e8742c80587dbc12c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:41:33.524919 6463 raft_consensus.cc:2229] T b32fa4a4ac5c45749a7f2f0859508c8a P b32dcc4d34fc44908dd3d879251d0ef0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:41:33.525476 6463 raft_consensus.cc:2258] T b32fa4a4ac5c45749a7f2f0859508c8a P b32dcc4d34fc44908dd3d879251d0ef0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:41:33.526510 6462 ts_tablet_manager.cc:1905] T b32fa4a4ac5c45749a7f2f0859508c8a P f758242b8e3d4b2e8742c80587dbc12c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:41:33.528167 6463 ts_tablet_manager.cc:1905] T b32fa4a4ac5c45749a7f2f0859508c8a P b32dcc4d34fc44908dd3d879251d0ef0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:41:33.529735 6461 tablet_replica.cc:330] T b32fa4a4ac5c45749a7f2f0859508c8a P f1e747c088b146c3a4eb96b53a980fb7: stopping tablet replica
I20260620 18:41:33.530140 6461 raft_consensus.cc:2229] T b32fa4a4ac5c45749a7f2f0859508c8a P f1e747c088b146c3a4eb96b53a980fb7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:41:33.530660 6461 raft_consensus.cc:2258] T b32fa4a4ac5c45749a7f2f0859508c8a P f1e747c088b146c3a4eb96b53a980fb7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:41:33.532878 6461 ts_tablet_manager.cc:1905] T b32fa4a4ac5c45749a7f2f0859508c8a P f1e747c088b146c3a4eb96b53a980fb7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:41:33.533790 6462 ts_tablet_manager.cc:1918] T b32fa4a4ac5c45749a7f2f0859508c8a P f758242b8e3d4b2e8742c80587dbc12c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:41:33.533957 6462 log.cc:1196] T b32fa4a4ac5c45749a7f2f0859508c8a P f758242b8e3d4b2e8742c80587dbc12c: Deleting WAL directory at /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-0/wal/wals/b32fa4a4ac5c45749a7f2f0859508c8a
I20260620 18:41:33.534478 6462 ts_tablet_manager.cc:1939] T b32fa4a4ac5c45749a7f2f0859508c8a P f758242b8e3d4b2e8742c80587dbc12c: Deleting consensus metadata
I20260620 18:41:33.535071 6436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:33.515133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33362 (local address 127.23.115.1:46031)
0620 18:41:33.515761 (+ 628us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:33.515768 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:33.515785 (+ 17us) server_negotiation.cc:407] Connection header received
0620 18:41:33.515843 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:33.515848 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:33.515925 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:33.516016 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:33.517985 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:33.519339 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:33.521415 (+ 2076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:33.522113 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:33.522238 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:33.526001 (+ 3763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:33.526022 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:33.526043 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:33.526105 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:33.530805 (+ 4700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:33.531741 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:33.531749 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:33.531752 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:33.531927 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:33.532444 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:33.532450 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:33.532452 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:33.532923 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:33.533256 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:33.534446 (+ 1190us) server_negotiation.cc:299] Negotiation successful
0620 18:41:33.534689 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":499}
I20260620 18:41:33.535705 6153 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:41:33.535879 6463 ts_tablet_manager.cc:1918] T b32fa4a4ac5c45749a7f2f0859508c8a P b32dcc4d34fc44908dd3d879251d0ef0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:41:33.536087 6463 log.cc:1196] T b32fa4a4ac5c45749a7f2f0859508c8a P b32dcc4d34fc44908dd3d879251d0ef0: Deleting WAL directory at /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-2/wal/wals/b32fa4a4ac5c45749a7f2f0859508c8a
I20260620 18:41:33.536320 5960 catalog_manager.cc:4841] TS f758242b8e3d4b2e8742c80587dbc12c (127.23.115.1:46031): tablet b32fa4a4ac5c45749a7f2f0859508c8a (table test-table [id=f634643bd89d484c948cd7a9b1051d9d]) successfully deleted
I20260620 18:41:33.536551 6463 ts_tablet_manager.cc:1939] T b32fa4a4ac5c45749a7f2f0859508c8a P b32dcc4d34fc44908dd3d879251d0ef0: Deleting consensus metadata
I20260620 18:41:33.538175 5960 catalog_manager.cc:4841] TS b32dcc4d34fc44908dd3d879251d0ef0 (127.23.115.3:41761): tablet b32fa4a4ac5c45749a7f2f0859508c8a (table test-table [id=f634643bd89d484c948cd7a9b1051d9d]) successfully deleted
I20260620 18:41:33.539958 6461 ts_tablet_manager.cc:1918] T b32fa4a4ac5c45749a7f2f0859508c8a P f1e747c088b146c3a4eb96b53a980fb7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:41:33.540200 6461 log.cc:1196] T b32fa4a4ac5c45749a7f2f0859508c8a P f1e747c088b146c3a4eb96b53a980fb7: Deleting WAL directory at /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-1/wal/wals/b32fa4a4ac5c45749a7f2f0859508c8a
I20260620 18:41:33.540763 6461 ts_tablet_manager.cc:1939] T b32fa4a4ac5c45749a7f2f0859508c8a P f1e747c088b146c3a4eb96b53a980fb7: Deleting consensus metadata
I20260620 18:41:33.542466 5958 catalog_manager.cc:4841] TS f1e747c088b146c3a4eb96b53a980fb7 (127.23.115.2:36887): tablet b32fa4a4ac5c45749a7f2f0859508c8a (table test-table [id=f634643bd89d484c948cd7a9b1051d9d]) successfully deleted
I20260620 18:41:33.631904 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 6010
I20260620 18:41:33.646909 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 6162
I20260620 18:41:33.660422 6434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:33.647773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34694 (local address 127.23.115.2:36887)
0620 18:41:33.647978 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:33.647984 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:33.648040 (+ 56us) server_negotiation.cc:407] Connection header received
0620 18:41:33.648110 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:33.648114 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:33.648177 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:33.648262 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:33.649622 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:33.650404 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:33.651822 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:33.652212 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:33.652317 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:33.654924 (+ 2607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:33.654964 (+ 40us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:33.654984 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:33.655027 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:33.657667 (+ 2640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:33.658377 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:33.658384 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:33.658387 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:33.658510 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:33.658870 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:33.658874 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:33.658876 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:33.659275 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:33.659541 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:33.659863 (+ 322us) server_negotiation.cc:299] Negotiation successful
0620 18:41:33.660073 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 18:41:33.661101 6285 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:41:33.748219 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 6162
I20260620 18:41:33.762230 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 6293
I20260620 18:41:33.775810 6432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:33.763055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50440 (local address 127.23.115.3:41761)
0620 18:41:33.763246 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:33.763251 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:33.763272 (+ 21us) server_negotiation.cc:407] Connection header received
0620 18:41:33.763331 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:33.763335 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:33.763398 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:33.763480 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:33.765025 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:33.765786 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:33.767606 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:33.768009 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:33.768105 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:33.770581 (+ 2476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:33.770594 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:33.770611 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:33.770653 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:33.773258 (+ 2605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:33.773926 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:33.773932 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:33.773934 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:33.774067 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:33.774421 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:33.774424 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:33.774427 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:33.774830 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:33.775102 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:33.775390 (+ 288us) server_negotiation.cc:299] Negotiation successful
0620 18:41:33.775551 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 18:41:33.776466 6416 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:41:33.865756 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 6293
I20260620 18:41:33.879179 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 5940
I20260620 18:41:33.880282 6003 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:41:33.992579 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 5940
I20260620 18:41:34.010414 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.115.62:35467
--webserver_interface=127.23.115.62
--webserver_port=45795
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab
--principal=kudu/127.23.115.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:33195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.115.62:35467
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:41:34.339381 6468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:41:34.339682 6468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:41:34.339802 6468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:41:34.347154 6468 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:41:34.347280 6468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:41:34.347349 6468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:41:34.347426 6468 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:41:34.347484 6468 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:41:34.358094 6468 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:33195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.115.62:35467
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab
--principal=kudu/127.23.115.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.115.62:35467
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.115.62
--webserver_port=45795
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:41:34.359470 6468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:41:34.361361 6468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:41:34.372928 6473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:34.372983 6474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:34.372983 6476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:41:34.373885 6468 server_base.cc:1034] running on GCE node
I20260620 18:41:34.374588 6468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:41:34.376014 6468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:41:34.377362 6468 hybrid_clock.cc:648] HybridClock initialized: now 1781980894377255 us; error 52 us; skew 500 ppm
I20260620 18:41:34.377907 6468 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:41:34 dist-test-slave-d645 krb5kdc[5389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980894, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:41:34.385932 6468 init.cc:377] Logged in from keytab as kudu/127.23.115.62@KRBTEST.COM (short username kudu)
I20260620 18:41:34.387547 6468 webserver.cc:460] Webserver started at http://127.23.115.62:45795/ using document root <none> and password file <none>
I20260620 18:41:34.388403 6468 fs_manager.cc:359] Metadata directory not provided
I20260620 18:41:34.388541 6468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:41:34.394032 6468 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 18:41:34.396955 6483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:34.398370 6468 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:41:34.398545 6468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/master-0/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/master-0/wal
uuid: "a15618092ac84abc82a0f318082d7455"
format_stamp: "Formatted at 2026-06-20 18:41:28 on dist-test-slave-d645"
server_key: "a4bfc05ff5542979236ba117c8b09d67"
server_key_iv: "18565b14a5c71413dce244e0ab893eac"
server_key_version: "encryptionkey@0"
I20260620 18:41:34.399547 6468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:41:34.417186 6468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:41:34.421238 6468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:41:34.421491 6468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:41:34.439380 6468 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.62:35467
I20260620 18:41:34.439388 6534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.62:35467 every 8 connection(s)
I20260620 18:41:34.440794 6468 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
I20260620 18:41:34.445329 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 6468
I20260620 18:41:34.446097 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.115.1:46031
--local_ip_for_outbound_sockets=127.23.115.1
--tserver_master_addrs=127.23.115.62:35467
--webserver_port=43901
--webserver_interface=127.23.115.1
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.1.keytab
--principal=kudu/127.23.115.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:33195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:41:34.447448 6535 sys_catalog.cc:263] Verifying existing consensus state
I20260620 18:41:34.450610 6535 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a15618092ac84abc82a0f318082d7455: Bootstrap starting.
I20260620 18:41:34.477874 6535 log.cc:824] T 00000000000000000000000000000000 P a15618092ac84abc82a0f318082d7455: Log is configured to *not* fsync() on all Append() calls
I20260620 18:41:34.496682 6535 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a15618092ac84abc82a0f318082d7455: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 18:41:34.497244 6535 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a15618092ac84abc82a0f318082d7455: Bootstrap complete.
I20260620 18:41:34.508471 6535 raft_consensus.cc:348] T 00000000000000000000000000000000 P a15618092ac84abc82a0f318082d7455 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a15618092ac84abc82a0f318082d7455" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 35467 } }
I20260620 18:41:34.509552 6535 raft_consensus.cc:729] T 00000000000000000000000000000000 P a15618092ac84abc82a0f318082d7455 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: a15618092ac84abc82a0f318082d7455, State: Initialized, Role: FOLLOWER
I20260620 18:41:34.510241 6535 consensus_queue.cc:260] T 00000000000000000000000000000000 P a15618092ac84abc82a0f318082d7455 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a15618092ac84abc82a0f318082d7455" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 35467 } }
I20260620 18:41:34.510469 6535 raft_consensus.cc:388] T 00000000000000000000000000000000 P a15618092ac84abc82a0f318082d7455 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:41:34.510582 6535 raft_consensus.cc:482] T 00000000000000000000000000000000 P a15618092ac84abc82a0f318082d7455 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:41:34.510751 6535 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a15618092ac84abc82a0f318082d7455 [term 2 FOLLOWER]: Advancing to term 3
I20260620 18:41:34.514879 6535 raft_consensus.cc:504] T 00000000000000000000000000000000 P a15618092ac84abc82a0f318082d7455 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a15618092ac84abc82a0f318082d7455" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 35467 } }
I20260620 18:41:34.515430 6535 leader_election.cc:302] T 00000000000000000000000000000000 P a15618092ac84abc82a0f318082d7455 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a15618092ac84abc82a0f318082d7455; no voters:
I20260620 18:41:34.516041 6535 leader_election.cc:288] T 00000000000000000000000000000000 P a15618092ac84abc82a0f318082d7455 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 18:41:34.516249 6539 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a15618092ac84abc82a0f318082d7455 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 18:41:34.517177 6539 raft_consensus.cc:686] T 00000000000000000000000000000000 P a15618092ac84abc82a0f318082d7455 [term 3 LEADER]: Becoming Leader. State: Replica: a15618092ac84abc82a0f318082d7455, State: Running, Role: LEADER
I20260620 18:41:34.517961 6535 sys_catalog.cc:564] T 00000000000000000000000000000000 P a15618092ac84abc82a0f318082d7455 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:41:34.517809 6539 consensus_queue.cc:237] T 00000000000000000000000000000000 P a15618092ac84abc82a0f318082d7455 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a15618092ac84abc82a0f318082d7455" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 35467 } }
I20260620 18:41:34.521111 6541 sys_catalog.cc:455] T 00000000000000000000000000000000 P a15618092ac84abc82a0f318082d7455 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a15618092ac84abc82a0f318082d7455. Latest consensus state: current_term: 3 leader_uuid: "a15618092ac84abc82a0f318082d7455" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a15618092ac84abc82a0f318082d7455" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 35467 } } }
I20260620 18:41:34.521643 6541 sys_catalog.cc:458] T 00000000000000000000000000000000 P a15618092ac84abc82a0f318082d7455 [sys.catalog]: This master's current role is: LEADER
I20260620 18:41:34.521224 6540 sys_catalog.cc:455] T 00000000000000000000000000000000 P a15618092ac84abc82a0f318082d7455 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "a15618092ac84abc82a0f318082d7455" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a15618092ac84abc82a0f318082d7455" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 35467 } } }
I20260620 18:41:34.521862 6540 sys_catalog.cc:458] T 00000000000000000000000000000000 P a15618092ac84abc82a0f318082d7455 [sys.catalog]: This master's current role is: LEADER
I20260620 18:41:34.544232 6546 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:41:34.553499 6546 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:41:34.555102 6546 catalog_manager.cc:1259] Loaded cluster ID: 2ce60297219545ebaf2008b08b4ce1ed
I20260620 18:41:34.555274 6546 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260620 18:41:34.557314 6546 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f50b66eb9ad google::LogMessage::Fail() at ??:0
@ 0x7f50b66ee88a google::LogMessage::SendToLog() at ??:0
@ 0x7f50b66eb4de google::LogMessage::Flush() at ??:0
@ 0x7f50b66eca19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f50cc11e512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f50cc11b976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f50cc1a3610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f50cc1a3439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f50b9337345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f50b933a91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f50b933a799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f50b9315eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f50bc5ef6db start_thread at ??:0
@ 0x7f50b558561f clone at ??:0
W20260620 18:41:34.866731 6537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:41:34.867029 6537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:41:34.867162 6537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:41:34.874709 6537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:41:34.874841 6537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:41:34.875049 6537 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.115.1
I20260620 18:41:34.885147 6537 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:33195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.1.keytab
--principal=kudu/127.23.115.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.115.1:46031
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.115.1
--webserver_port=43901
--tserver_master_addrs=127.23.115.62:35467
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.115.1
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:41:34.886391 6537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:41:34.887686 6537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:41:34.900279 6577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:34.900309 6578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:34.900411 6580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:41:34.901036 6537 server_base.cc:1034] running on GCE node
I20260620 18:41:34.901567 6537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:41:34.902462 6537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:41:34.903671 6537 hybrid_clock.cc:648] HybridClock initialized: now 1781980894903624 us; error 35 us; skew 500 ppm
I20260620 18:41:34.903985 6537 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:41:34 dist-test-slave-d645 krb5kdc[5389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980894, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:41:34.907136 6537 init.cc:377] Logged in from keytab as kudu/127.23.115.1@KRBTEST.COM (short username kudu)
I20260620 18:41:34.908586 6537 webserver.cc:460] Webserver started at http://127.23.115.1:43901/ using document root <none> and password file <none>
I20260620 18:41:34.909230 6537 fs_manager.cc:359] Metadata directory not provided
I20260620 18:41:34.909348 6537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:41:34.914824 6537 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260620 18:41:34.917730 6587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:34.919227 6537 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:41:34.919415 6537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-0/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-0/wal
uuid: "f758242b8e3d4b2e8742c80587dbc12c"
format_stamp: "Formatted at 2026-06-20 18:41:29 on dist-test-slave-d645"
server_key: "e7283e95ed8c97ee2dd7a36dcede1350"
server_key_iv: "38f50c2f96fc434ee8cafffd757f8086"
server_key_version: "encryptionkey@0"
I20260620 18:41:34.920406 6537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:41:34.953462 6537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:41:34.957535 6537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:41:34.957808 6537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:41:34.958801 6537 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:41:34.960163 6537 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:41:34.960270 6537 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:34.960379 6537 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:41:34.960451 6537 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:34.992168 6537 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.1:46031
I20260620 18:41:34.992223 6699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.1:46031 every 8 connection(s)
I20260620 18:41:34.993551 6537 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-0/data/info.pb
I20260620 18:41:34.998318 6702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:34.996520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:35467 (local address 127.23.115.1:34749)
0620 18:41:34.997314 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:34.997538 (+ 224us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:35467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":301,"threads_started":1}
W20260620 18:41:34.998916 6700 heartbeater.cc:644] Failed to heartbeat to 127.23.115.62:35467 (0 consecutive failures): Network error: Failed to ping master at 127.23.115.62:35467: Client connection negotiation failed: client connection to 127.23.115.62:35467: connect: Connection refused (error 111)
I20260620 18:41:34.999732 6702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:34.999423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:35467 (local address 127.23.115.1:36495)
0620 18:41:34.999559 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:34.999631 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:35467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 18:41:35.000504 6702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:35.000200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:35467 (local address 127.23.115.1:39887)
0620 18:41:35.000333 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:35.000403 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:35467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 18:41:35.000720 6700 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 18:41:35.003803 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 6537
I20260620 18:41:35.004390 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.115.2:36887
--local_ip_for_outbound_sockets=127.23.115.2
--tserver_master_addrs=127.23.115.62:35467
--webserver_port=41981
--webserver_interface=127.23.115.2
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.2.keytab
--principal=kudu/127.23.115.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:33195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:41:35.345700 6703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:41:35.346112 6703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:41:35.346288 6703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:41:35.355866 6703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:41:35.355984 6703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:41:35.356096 6703 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.115.2
I20260620 18:41:35.365800 6703 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:33195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.2.keytab
--principal=kudu/127.23.115.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.115.2:36887
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.115.2
--webserver_port=41981
--tserver_master_addrs=127.23.115.62:35467
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.115.2
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:41:35.367059 6703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:41:35.368292 6703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:41:35.380815 6708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:35.380985 6711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:35.381011 6709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:41:35.382078 6703 server_base.cc:1034] running on GCE node
I20260620 18:41:35.382647 6703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:41:35.383651 6703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:41:35.385190 6703 hybrid_clock.cc:648] HybridClock initialized: now 1781980895385027 us; error 37 us; skew 500 ppm
I20260620 18:41:35.385744 6703 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:41:35 dist-test-slave-d645 krb5kdc[5389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980895, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:41:35.389521 6703 init.cc:377] Logged in from keytab as kudu/127.23.115.2@KRBTEST.COM (short username kudu)
I20260620 18:41:35.391037 6703 webserver.cc:460] Webserver started at http://127.23.115.2:41981/ using document root <none> and password file <none>
I20260620 18:41:35.391683 6703 fs_manager.cc:359] Metadata directory not provided
I20260620 18:41:35.391786 6703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:41:35.396941 6703 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260620 18:41:35.399708 6718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:35.400991 6703 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:41:35.401142 6703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-1/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-1/wal
uuid: "f1e747c088b146c3a4eb96b53a980fb7"
format_stamp: "Formatted at 2026-06-20 18:41:29 on dist-test-slave-d645"
server_key: "c0a8216d3a3a482d534a45555a5a35b1"
server_key_iv: "bb1d71dc245e2bfd52b199f6f4c7a19d"
server_key_version: "encryptionkey@0"
I20260620 18:41:35.402097 6703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:41:35.423502 6703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:41:35.427932 6703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:41:35.428303 6703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:41:35.429440 6703 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:41:35.430858 6703 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:41:35.431054 6703 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260620 18:41:35.431186 6703 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:41:35.431288 6703 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:35.462998 6703 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.2:36887
I20260620 18:41:35.463042 6830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.2:36887 every 8 connection(s)
I20260620 18:41:35.464390 6703 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-1/data/info.pb
I20260620 18:41:35.468974 6833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:35.467234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:35467 (local address 127.23.115.2:54811)
0620 18:41:35.467941 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:35.468165 (+ 224us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:35467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":259,"threads_started":1}
W20260620 18:41:35.469717 6831 heartbeater.cc:644] Failed to heartbeat to 127.23.115.62:35467 (0 consecutive failures): Network error: Failed to ping master at 127.23.115.62:35467: Client connection negotiation failed: client connection to 127.23.115.62:35467: connect: Connection refused (error 111)
I20260620 18:41:35.469738 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 6703
I20260620 18:41:35.470400 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.115.3:41761
--local_ip_for_outbound_sockets=127.23.115.3
--tserver_master_addrs=127.23.115.62:35467
--webserver_port=43223
--webserver_interface=127.23.115.3
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.3.keytab
--principal=kudu/127.23.115.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:33195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:41:35.470582 6833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:35.470214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:35467 (local address 127.23.115.2:50439)
0620 18:41:35.470382 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:35.470448 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:35467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260620 18:41:35.471418 6833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:35.471087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:35467 (local address 127.23.115.2:59799)
0620 18:41:35.471235 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:35.471313 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:35467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260620 18:41:35.471647 6831 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 18:41:35.783821 6834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:41:35.784183 6834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:41:35.784335 6834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:41:35.792551 6834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:41:35.792685 6834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:41:35.792838 6834 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.115.3
I20260620 18:41:35.802863 6834 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:33195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.3.keytab
--principal=kudu/127.23.115.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.115.3:41761
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.115.3
--webserver_port=43223
--tserver_master_addrs=127.23.115.62:35467
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.115.3
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:41:35.804173 6834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:41:35.805490 6834 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:41:35.819110 6842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:35.819207 6839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:35.819191 6840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:41:35.820154 6834 server_base.cc:1034] running on GCE node
I20260620 18:41:35.820786 6834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:41:35.821797 6834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:41:35.823138 6834 hybrid_clock.cc:648] HybridClock initialized: now 1781980895823082 us; error 55 us; skew 500 ppm
I20260620 18:41:35.823520 6834 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:41:35 dist-test-slave-d645 krb5kdc[5389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980895, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:41:35.826889 6834 init.cc:377] Logged in from keytab as kudu/127.23.115.3@KRBTEST.COM (short username kudu)
I20260620 18:41:35.828594 6834 webserver.cc:460] Webserver started at http://127.23.115.3:43223/ using document root <none> and password file <none>
I20260620 18:41:35.829309 6834 fs_manager.cc:359] Metadata directory not provided
I20260620 18:41:35.829428 6834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:41:35.835433 6834 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 18:41:35.838243 6849 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:35.839748 6834 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:41:35.839912 6834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-2/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-2/wal
uuid: "b32dcc4d34fc44908dd3d879251d0ef0"
format_stamp: "Formatted at 2026-06-20 18:41:30 on dist-test-slave-d645"
server_key: "3adba41137f35f38872966a1dc3cd1e6"
server_key_iv: "6b0c1a0c31be04487a2a6d029ab1eea9"
server_key_version: "encryptionkey@0"
I20260620 18:41:35.840895 6834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:41:35.859030 6834 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:41:35.864684 6834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:41:35.864969 6834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:41:35.865955 6834 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:41:35.867300 6834 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:41:35.867394 6834 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:35.867501 6834 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:41:35.867568 6834 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:35.902009 6834 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.3:41761
I20260620 18:41:35.902063 6961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.3:41761 every 8 connection(s)
I20260620 18:41:35.903427 6834 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980676253068-24012-0/minicluster-data/ts-2/data/info.pb
I20260620 18:41:35.908206 6964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:35.906395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:35467 (local address 127.23.115.3:43421)
0620 18:41:35.907176 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:35.907393 (+ 217us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:35467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":313,"threads_started":1}
W20260620 18:41:35.908951 6962 heartbeater.cc:644] Failed to heartbeat to 127.23.115.62:35467 (0 consecutive failures): Network error: Failed to ping master at 127.23.115.62:35467: Client connection negotiation failed: client connection to 127.23.115.62:35467: connect: Connection refused (error 111)
I20260620 18:41:35.910089 6964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:35.909435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:35467 (local address 127.23.115.3:43559)
0620 18:41:35.909727 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:35.909968 (+ 241us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:35467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260620 18:41:35.911265 6964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:35.910827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:35467 (local address 127.23.115.3:52063)
0620 18:41:35.911037 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:35.911123 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:35467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260620 18:41:35.911530 6962 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 18:41:35.911849 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 6834
I20260620 18:41:36.002152 6966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:36.001268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:35467 (local address 127.23.115.1:45077)
0620 18:41:36.001743 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:36.001845 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:35467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":226,"threads_started":1}
I20260620 18:41:36.472980 6967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:36.472197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:35467 (local address 127.23.115.2:58551)
0620 18:41:36.472649 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:36.472748 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:35467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":207,"threads_started":1}
I20260620 18:41:36.912958 6968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:36.912118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:35467 (local address 127.23.115.3:54659)
0620 18:41:36.912595 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:36.912710 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:35467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":229,"threads_started":1}
I20260620 18:41:37.004479 6969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:37.003499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:35467 (local address 127.23.115.1:52243)
0620 18:41:37.004195 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:37.004302 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:35467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":333,"threads_started":1}
I20260620 18:41:37.475332 6970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:37.474285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:35467 (local address 127.23.115.2:49799)
0620 18:41:37.475037 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:37.475142 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:35467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":366,"threads_started":1}
I20260620 18:41:37.914793 6971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:37.913889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:35467 (local address 127.23.115.3:60175)
0620 18:41:37.914516 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:37.914611 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:35467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":292,"threads_started":1}
I20260620 18:41:38.006611 6972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:38.005501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:35467 (local address 127.23.115.1:45547)
0620 18:41:38.006168 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:38.006357 (+ 189us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:35467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":323,"threads_started":1}
I20260620 18:41:38.477466 6973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:38.476344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:35467 (local address 127.23.115.2:58697)
0620 18:41:38.477179 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:38.477272 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:35467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":727,"thread_start_us":346,"threads_started":1}
I20260620 18:41:38.917099 6974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:38.916017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:35467 (local address 127.23.115.3:33595)
0620 18:41:38.916727 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:38.916865 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:35467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":427,"threads_started":1}
I20260620 18:41:39.008754 6975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:39.007668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:35467 (local address 127.23.115.1:57423)
0620 18:41:39.008385 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:39.008517 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:35467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":381,"threads_started":1}
I20260620 18:41:39.479996 6976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:39.478838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:35467 (local address 127.23.115.2:56389)
0620 18:41:39.479666 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:39.479783 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:35467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":669,"thread_start_us":363,"threads_started":1}
I20260620 18:41:39.919584 6977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:39.918504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:35467 (local address 127.23.115.3:56663)
0620 18:41:39.919158 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:39.919278 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:35467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":304,"threads_started":1}
I20260620 18:41:40.011361 6978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:40.010315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:35467 (local address 127.23.115.1:37033)
0620 18:41:40.011001 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:40.011151 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:35467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":330,"threads_started":1}
I20260620 18:41:40.481873 6979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:40.481108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:35467 (local address 127.23.115.2:32781)
0620 18:41:40.481607 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:40.481696 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:35467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":251,"threads_started":1}
I20260620 18:41:40.921485 6980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:40.920678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:35467 (local address 127.23.115.3:53553)
0620 18:41:40.921173 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:40.921295 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:35467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":258,"threads_started":1}
I20260620 18:41:41.013235 6981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:41.012452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:35467 (local address 127.23.115.1:57117)
0620 18:41:41.012935 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:41.013047 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:35467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":244,"threads_started":1}
I20260620 18:41:41.483608 6982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:41.482800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:35467 (local address 127.23.115.2:46429)
0620 18:41:41.483298 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:41.483404 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:35467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":258,"threads_started":1}
I20260620 18:41:41.923892 6983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:41.922633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:35467 (local address 127.23.115.3:52963)
0620 18:41:41.923419 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:41.923648 (+ 229us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:35467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":383,"threads_started":1}
I20260620 18:41:42.015323 6984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:42.014448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:35467 (local address 127.23.115.1:33951)
0620 18:41:42.015015 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:42.015107 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:35467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":238,"threads_started":1}
I20260620 18:41:42.485419 6985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:42.484500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:35467 (local address 127.23.115.2:50661)
0620 18:41:42.485080 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:42.485199 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:35467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":302,"threads_started":1}
I20260620 18:41:42.925880 6986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:42.925037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:35467 (local address 127.23.115.3:53623)
0620 18:41:42.925542 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:42.925635 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:35467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":223,"threads_started":1}
I20260620 18:41:43.017313 6987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:43.016547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:35467 (local address 127.23.115.1:44017)
0620 18:41:43.017061 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:43.017151 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:35467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":267,"threads_started":1}
I20260620 18:41:43.487123 6988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:43.486327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:35467 (local address 127.23.115.2:42531)
0620 18:41:43.486824 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:43.486921 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:35467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":257,"threads_started":1}
I20260620 18:41:43.927513 6989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:43.926786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:35467 (local address 127.23.115.3:34199)
0620 18:41:43.927270 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:43.927348 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:35467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":260,"threads_started":1}
I20260620 18:41:44.019608 6990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:44.018610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:35467 (local address 127.23.115.1:52991)
0620 18:41:44.019284 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:44.019434 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:35467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":380,"threads_started":1}
I20260620 18:41:44.488847 6991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:44.488074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:35467 (local address 127.23.115.2:58287)
0620 18:41:44.488579 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:44.488669 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:35467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":256,"threads_started":1}
I20260620 18:41:44.929241 6992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:44.928453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:35467 (local address 127.23.115.3:46205)
0620 18:41:44.928986 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:44.929069 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:35467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":276,"threads_started":1}
I20260620 18:41:45.022233 6993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:45.021254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:35467 (local address 127.23.115.1:51893)
0620 18:41:45.021955 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:45.022049 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:35467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":360,"threads_started":1}
I20260620 18:41:45.490536 6994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:45.489803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:35467 (local address 127.23.115.2:34373)
0620 18:41:45.490290 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:45.490378 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:35467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":270,"threads_started":1}
I20260620 18:41:45.931010 6995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:45.930221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:35467 (local address 127.23.115.3:51407)
0620 18:41:45.930722 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:45.930808 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:35467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":277,"threads_started":1}
I20260620 18:41:46.023944 6996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:46.023232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:35467 (local address 127.23.115.1:54579)
0620 18:41:46.023677 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:46.023771 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:35467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":218,"threads_started":1}
I20260620 18:41:46.492492 6997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:46.491631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:35467 (local address 127.23.115.2:59093)
0620 18:41:46.492189 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:46.492304 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:35467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":279,"threads_started":1}
I20260620 18:41:46.932716 6998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:46.931984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:35467 (local address 127.23.115.3:43839)
0620 18:41:46.932430 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:46.932523 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:35467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":229,"threads_started":1}
I20260620 18:41:47.025651 6999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:47.024949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:35467 (local address 127.23.115.1:53671)
0620 18:41:47.025415 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:47.025495 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:35467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":241,"threads_started":1}
I20260620 18:41:47.494459 7000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:47.493712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:35467 (local address 127.23.115.2:54173)
0620 18:41:47.494203 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:47.494297 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:35467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":254,"threads_started":1}
I20260620 18:41:47.934435 7001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:47.933704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:35467 (local address 127.23.115.3:47057)
0620 18:41:47.934188 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:47.934274 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:35467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":250,"threads_started":1}
I20260620 18:41:48.027547 7002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:48.026598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:35467 (local address 127.23.115.1:55395)
0620 18:41:48.027258 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:48.027358 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:35467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":262,"threads_started":1}
I20260620 18:41:48.496243 7003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:48.495429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:35467 (local address 127.23.115.2:49077)
0620 18:41:48.495946 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:48.496050 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:35467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":256,"threads_started":1}
I20260620 18:41:48.936313 7004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:48.935448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:35467 (local address 127.23.115.3:34425)
0620 18:41:48.936012 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:48.936106 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:35467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":314,"threads_started":1}
I20260620 18:41:49.029834 7005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:49.028880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:35467 (local address 127.23.115.1:53663)
0620 18:41:49.029511 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:49.029632 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:35467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":330,"threads_started":1}
I20260620 18:41:49.497889 7006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:49.497156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:35467 (local address 127.23.115.2:39265)
0620 18:41:49.497616 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:49.497695 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:35467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":245,"threads_started":1}
I20260620 18:41:49.937919 7007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:49.937221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:35467 (local address 127.23.115.3:51899)
0620 18:41:49.937644 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:49.937740 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:35467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":213,"threads_started":1}
I20260620 18:41:50.032059 7008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:50.031239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:35467 (local address 127.23.115.1:39015)
0620 18:41:50.031758 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:50.031859 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:35467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":272,"threads_started":1}
I20260620 18:41:50.499905 7009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:50.499133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:35467 (local address 127.23.115.2:41193)
0620 18:41:50.499612 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:50.499718 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:35467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":266,"threads_started":1}
I20260620 18:41:50.916847 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 6537
I20260620 18:41:50.940009 7011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:50.939244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:35467 (local address 127.23.115.3:40837)
0620 18:41:50.939720 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:50.939805 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:35467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":257,"threads_started":1}
I20260620 18:41:50.944574 7010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:50.918375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43564 (local address 127.23.115.1:46031)
0620 18:41:50.919277 (+ 902us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:50.919298 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:50.919351 (+ 53us) server_negotiation.cc:407] Connection header received
0620 18:41:50.920979 (+ 1628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:50.920988 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:50.921762 (+ 774us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:50.922339 (+ 577us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:50.924628 (+ 2289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:50.926318 (+ 1690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:50.927919 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:50.928963 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:50.929323 (+ 360us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:50.937122 (+ 7799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:50.937133 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:50.937151 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:50.937201 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:50.940566 (+ 3365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:50.941355 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:50.941362 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:50.941369 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:50.941587 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:50.942195 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:50.942200 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:50.942202 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:50.942827 (+ 625us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:50.943143 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:50.943701 (+ 558us) server_negotiation.cc:299] Negotiation successful
0620 18:41:50.944196 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":684,"thread_start_us":582,"threads_started":1}
I20260620 18:41:50.946033 6695 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:41:51.035838 7013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:51.032998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:35467 (local address 127.23.115.1:53103)
0620 18:41:51.035394 (+ 2396us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:51.035586 (+ 192us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:35467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2233,"thread_start_us":2096,"threads_started":1}
I20260620 18:41:51.038698 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 6537
I20260620 18:41:51.054329 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 6703
I20260620 18:41:51.072098 7014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:51.055329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53032 (local address 127.23.115.2:36887)
0620 18:41:51.055868 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:51.055882 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:51.055920 (+ 38us) server_negotiation.cc:407] Connection header received
0620 18:41:51.057542 (+ 1622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:51.057552 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:51.057986 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:51.058290 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:51.059805 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:51.061037 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:51.062286 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:51.063003 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:51.063140 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:51.065976 (+ 2836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:51.065984 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:51.066002 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:51.066042 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:51.069126 (+ 3084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:51.069777 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:51.069784 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:51.069791 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:51.069958 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:51.070281 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:51.070284 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:51.070286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:51.070794 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:51.071072 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:51.071427 (+ 355us) server_negotiation.cc:299] Negotiation successful
0620 18:41:51.071816 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":245,"threads_started":1}
I20260620 18:41:51.073462 6826 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:41:51.158258 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 6703
I20260620 18:41:51.173066 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 6834
I20260620 18:41:51.190490 7016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:51.174224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44976 (local address 127.23.115.3:41761)
0620 18:41:51.174841 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:51.174855 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:51.174897 (+ 42us) server_negotiation.cc:407] Connection header received
0620 18:41:51.176559 (+ 1662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:51.176567 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:51.177023 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:51.177338 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:51.178797 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:51.180091 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:51.181309 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:51.181954 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:51.182115 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:51.184617 (+ 2502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:51.184624 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:51.184639 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:51.184676 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:51.187450 (+ 2774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:51.188071 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:51.188077 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:51.188083 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:51.188243 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:51.188592 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:51.188595 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:51.188597 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:51.189100 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:51.189382 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:51.189787 (+ 405us) server_negotiation.cc:299] Negotiation successful
0620 18:41:51.190191 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":268,"threads_started":1}
I20260620 18:41:51.191959 6957 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:41:51.275820 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 6834
2026-06-20T18:41:51Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24034 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:41:51 dist-test-slave-d645 krb5kdc[7020](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:41:51 dist-test-slave-d645 krb5kdc[7020](info): set up 2 sockets
Jun 20 18:41:51 dist-test-slave-d645 krb5kdc[7020](info): commencing operation
krb5kdc: starting...
W20260620 18:41:52.307564 24012 mini_kdc.cc:121] Time spent starting KDC: real 0.994s user 0.008s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/krb5kdc/test-user.keytab.
Jun 20 18:41:52 dist-test-slave-d645 krb5kdc[7020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980912, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:41:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:41:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.115.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.115.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.115.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab.
Entry for principal kudu/127.23.115.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab.
I20260620 18:41:52.534780 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.115.62:44779
--webserver_interface=127.23.115.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab
--principal=kudu/127.23.115.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:41715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.115.62:44779
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:41:52.862669 7033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:41:52.862977 7033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:41:52.863092 7033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:41:52.870398 7033 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 18:41:52.870491 7033 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 18:41:52.870527 7033 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 18:41:52.870576 7033 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:41:52.870632 7033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:41:52.870697 7033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:41:52.870739 7033 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:41:52.870790 7033 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:41:52.880489 7033 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:41715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.23.115.62:44779
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab
--principal=kudu/127.23.115.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.115.62:44779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.115.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:41:52.881673 7033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:41:52.882956 7033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:41:52.894235 7038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:52.894274 7041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:52.894348 7039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:41:52.895232 7033 server_base.cc:1034] running on GCE node
I20260620 18:41:52.896000 7033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:41:52.897984 7033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:41:52.899219 7033 hybrid_clock.cc:648] HybridClock initialized: now 1781980912899142 us; error 42 us; skew 500 ppm
I20260620 18:41:52.899564 7033 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:41:52 dist-test-slave-d645 krb5kdc[7020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980912, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:41:52.902726 7033 init.cc:377] Logged in from keytab as kudu/127.23.115.62@KRBTEST.COM (short username kudu)
I20260620 18:41:52.904521 7033 webserver.cc:460] Webserver started at http://127.23.115.62:36309/ using document root <none> and password file <none>
I20260620 18:41:52.905272 7033 fs_manager.cc:359] Metadata directory not provided
I20260620 18:41:52.905375 7033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:41:52.905691 7033 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:41:52.908743 7033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/master-0/data/instance:
uuid: "2ea2131d7e994f00a8a2e0f24724e171"
format_stamp: "Formatted at 2026-06-20 18:41:52 on dist-test-slave-d645"
server_key: "d89fe0e68734db40a03bc14e40aaf1c6"
server_key_iv: "8b96b28491fa7104a042e84a15200d10"
server_key_version: "encryptionkey@0"
I20260620 18:41:52.909513 7033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/master-0/wal/instance:
uuid: "2ea2131d7e994f00a8a2e0f24724e171"
format_stamp: "Formatted at 2026-06-20 18:41:52 on dist-test-slave-d645"
server_key: "d89fe0e68734db40a03bc14e40aaf1c6"
server_key_iv: "8b96b28491fa7104a042e84a15200d10"
server_key_version: "encryptionkey@0"
I20260620 18:41:52.914549 7033 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260620 18:41:52.918108 7048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:52.919906 7033 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 18:41:52.920081 7033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/master-0/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/master-0/wal
uuid: "2ea2131d7e994f00a8a2e0f24724e171"
format_stamp: "Formatted at 2026-06-20 18:41:52 on dist-test-slave-d645"
server_key: "d89fe0e68734db40a03bc14e40aaf1c6"
server_key_iv: "8b96b28491fa7104a042e84a15200d10"
server_key_version: "encryptionkey@0"
I20260620 18:41:52.920291 7033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:41:52.932408 7033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:41:52.936458 7033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:41:52.936722 7033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:41:52.954653 7033 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.62:44779
I20260620 18:41:52.954628 7099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.62:44779 every 8 connection(s)
I20260620 18:41:52.956041 7033 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
I20260620 18:41:52.959582 7100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:41:52.966338 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 7033
I20260620 18:41:52.966575 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/master-0/wal/instance
I20260620 18:41:52.967398 24012 external_mini_cluster.cc:1428] Setting key f2b5caccad1ef16a8a11eb646a80dbec
I20260620 18:41:52.970388 7100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ea2131d7e994f00a8a2e0f24724e171: Bootstrap starting.
I20260620 18:41:52.973744 7100 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ea2131d7e994f00a8a2e0f24724e171: Neither blocks nor log segments found. Creating new log.
I20260620 18:41:52.974980 7100 log.cc:824] T 00000000000000000000000000000000 P 2ea2131d7e994f00a8a2e0f24724e171: Log is configured to *not* fsync() on all Append() calls
I20260620 18:41:52.978288 7100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ea2131d7e994f00a8a2e0f24724e171: No bootstrap required, opened a new log
Jun 20 18:41:52 dist-test-slave-d645 krb5kdc[7020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980912, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.115.62@KRBTEST.COM
I20260620 18:41:52.988404 7100 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ea2131d7e994f00a8a2e0f24724e171 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ea2131d7e994f00a8a2e0f24724e171" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 44779 } }
I20260620 18:41:52.989099 7100 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2ea2131d7e994f00a8a2e0f24724e171 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:41:52.989253 7100 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ea2131d7e994f00a8a2e0f24724e171 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ea2131d7e994f00a8a2e0f24724e171, State: Initialized, Role: FOLLOWER
I20260620 18:41:52.990011 7100 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ea2131d7e994f00a8a2e0f24724e171 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ea2131d7e994f00a8a2e0f24724e171" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 44779 } }
I20260620 18:41:52.990306 7100 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2ea2131d7e994f00a8a2e0f24724e171 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:41:52.990455 7100 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ea2131d7e994f00a8a2e0f24724e171 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:41:52.990633 7100 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ea2131d7e994f00a8a2e0f24724e171 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:41:52.992791 7103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:52.969444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33098 (local address 127.23.115.62:44779)
0620 18:41:52.970161 (+ 717us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:52.970172 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:52.970214 (+ 42us) server_negotiation.cc:407] Connection header received
0620 18:41:52.971950 (+ 1736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:52.971974 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:52.972348 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:52.972659 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:52.974503 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:52.976274 (+ 1771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:52.978186 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:52.978993 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:52.979134 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:52.984745 (+ 5611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:52.984757 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:52.984775 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:52.984817 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:52.988059 (+ 3242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:52.989143 (+ 1084us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:52.989153 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:52.989162 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:52.989410 (+ 248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:52.989888 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:52.989891 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:52.989893 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:52.990520 (+ 627us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:52.990790 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:52.991231 (+ 441us) server_negotiation.cc:299] Negotiation successful
0620 18:41:52.991652 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":314,"threads_started":1}
I20260620 18:41:52.993541 7100 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ea2131d7e994f00a8a2e0f24724e171 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ea2131d7e994f00a8a2e0f24724e171" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 44779 } }
I20260620 18:41:52.994045 7100 leader_election.cc:302] T 00000000000000000000000000000000 P 2ea2131d7e994f00a8a2e0f24724e171 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ea2131d7e994f00a8a2e0f24724e171; no voters:
I20260620 18:41:52.994633 7100 leader_election.cc:288] T 00000000000000000000000000000000 P 2ea2131d7e994f00a8a2e0f24724e171 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:41:52.994786 7105 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2ea2131d7e994f00a8a2e0f24724e171 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:41:52.996726 7100 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ea2131d7e994f00a8a2e0f24724e171 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:41:53.001441 7105 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2ea2131d7e994f00a8a2e0f24724e171 [term 1 LEADER]: Becoming Leader. State: Replica: 2ea2131d7e994f00a8a2e0f24724e171, State: Running, Role: LEADER
I20260620 18:41:53.002058 7105 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ea2131d7e994f00a8a2e0f24724e171 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ea2131d7e994f00a8a2e0f24724e171" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 44779 } }
I20260620 18:41:53.005261 7107 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ea2131d7e994f00a8a2e0f24724e171 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ea2131d7e994f00a8a2e0f24724e171. Latest consensus state: current_term: 1 leader_uuid: "2ea2131d7e994f00a8a2e0f24724e171" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ea2131d7e994f00a8a2e0f24724e171" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 44779 } } }
I20260620 18:41:53.005944 7107 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ea2131d7e994f00a8a2e0f24724e171 [sys.catalog]: This master's current role is: LEADER
I20260620 18:41:53.005499 7106 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ea2131d7e994f00a8a2e0f24724e171 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2ea2131d7e994f00a8a2e0f24724e171" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ea2131d7e994f00a8a2e0f24724e171" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 44779 } } }
I20260620 18:41:53.007200 7106 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ea2131d7e994f00a8a2e0f24724e171 [sys.catalog]: This master's current role is: LEADER
I20260620 18:41:53.008493 7120 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:41:53.012661 7120 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:41:53.024317 7120 catalog_manager.cc:1347] Generated new cluster ID: a10c99f42da74d3bbf169911bfe9f448
I20260620 18:41:53.024459 7120 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:41:53.052784 7120 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:41:53.054303 7120 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:41:53.062855 7120 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2ea2131d7e994f00a8a2e0f24724e171: Generated new TSK 0
I20260620 18:41:53.063889 7120 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.115.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.115.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.115.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.1.keytab.
Entry for principal kudu/127.23.115.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.1.keytab.
I20260620 18:41:53.118710 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.115.1:0
--local_ip_for_outbound_sockets=127.23.115.1
--webserver_interface=127.23.115.1
--webserver_port=0
--tserver_master_addrs=127.23.115.62:44779
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.1.keytab
--principal=kudu/127.23.115.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:41715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:41:53.451256 7126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:41:53.451534 7126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:41:53.451628 7126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:41:53.458703 7126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:41:53.458812 7126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:41:53.458945 7126 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.115.1
I20260620 18:41:53.468525 7126 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:41715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.1.keytab
--principal=kudu/127.23.115.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.115.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.115.1
--webserver_port=0
--tserver_master_addrs=127.23.115.62:44779
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.115.1
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:41:53.469800 7126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:41:53.471031 7126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:41:53.483275 7132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:53.483353 7131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:53.483294 7134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:41:53.484499 7126 server_base.cc:1034] running on GCE node
I20260620 18:41:53.485076 7126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:41:53.485996 7126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:41:53.487260 7126 hybrid_clock.cc:648] HybridClock initialized: now 1781980913487166 us; error 37 us; skew 500 ppm
I20260620 18:41:53.487569 7126 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:41:53 dist-test-slave-d645 krb5kdc[7020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980913, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:41:53.490592 7126 init.cc:377] Logged in from keytab as kudu/127.23.115.1@KRBTEST.COM (short username kudu)
I20260620 18:41:53.492647 7126 webserver.cc:460] Webserver started at http://127.23.115.1:35971/ using document root <none> and password file <none>
I20260620 18:41:53.493319 7126 fs_manager.cc:359] Metadata directory not provided
I20260620 18:41:53.493460 7126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:41:53.493750 7126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:41:53.496737 7126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-0/data/instance:
uuid: "86629c1da49b4263ae52078ef6a60192"
format_stamp: "Formatted at 2026-06-20 18:41:53 on dist-test-slave-d645"
server_key: "a45a2eff8f0bc3ba6e9dc818944ad5ae"
server_key_iv: "7ae3c4659b337945e16c951f830ad5b9"
server_key_version: "encryptionkey@0"
I20260620 18:41:53.497488 7126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-0/wal/instance:
uuid: "86629c1da49b4263ae52078ef6a60192"
format_stamp: "Formatted at 2026-06-20 18:41:53 on dist-test-slave-d645"
server_key: "a45a2eff8f0bc3ba6e9dc818944ad5ae"
server_key_iv: "7ae3c4659b337945e16c951f830ad5b9"
server_key_version: "encryptionkey@0"
I20260620 18:41:53.502619 7126 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.006s
I20260620 18:41:53.506657 7141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:53.508466 7126 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260620 18:41:53.508634 7126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-0/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-0/wal
uuid: "86629c1da49b4263ae52078ef6a60192"
format_stamp: "Formatted at 2026-06-20 18:41:53 on dist-test-slave-d645"
server_key: "a45a2eff8f0bc3ba6e9dc818944ad5ae"
server_key_iv: "7ae3c4659b337945e16c951f830ad5b9"
server_key_version: "encryptionkey@0"
I20260620 18:41:53.508833 7126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:41:53.537341 7126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:41:53.541177 7126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:41:53.541433 7126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:41:53.542344 7126 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:41:53.543646 7126 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:41:53.543722 7126 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:53.543851 7126 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:41:53.543906 7126 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:53.576889 7126 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.1:46605
I20260620 18:41:53.576948 7253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.1:46605 every 8 connection(s)
I20260620 18:41:53.578182 7126 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-0/data/info.pb
I20260620 18:41:53.581789 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 7126
I20260620 18:41:53.582011 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-0/wal/instance
I20260620 18:41:53.582798 24012 external_mini_cluster.cc:1428] Setting key 8e7004d5a521e99044b7e232be60ff84
Jun 20 18:41:53 dist-test-slave-d645 krb5kdc[7020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980913, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.1@KRBTEST.COM for kudu/127.23.115.62@KRBTEST.COM
I20260620 18:41:53.597669 7256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:53.580658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.1:48115 (local address 127.23.115.62:44779)
0620 18:41:53.581127 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:53.581131 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:53.582241 (+ 1110us) server_negotiation.cc:407] Connection header received
0620 18:41:53.584301 (+ 2060us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:53.584305 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:53.584368 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:53.584484 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:53.585975 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:53.586687 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:53.588227 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:53.588604 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:53.588713 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:53.591913 (+ 3200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:53.591918 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:53.591921 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:53.591951 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:53.594557 (+ 2606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:53.595414 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:53.595419 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:53.595422 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:53.595546 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:53.596012 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:53.596015 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:53.596017 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:53.596293 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:53.596432 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:53.597199 (+ 767us) server_negotiation.cc:299] Negotiation successful
0620 18:41:53.597380 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":231,"threads_started":1}
I20260620 18:41:53.598910 7257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:53.581157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:44779 (local address 127.23.115.1:48115)
0620 18:41:53.582093 (+ 936us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:53.582140 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:53.584029 (+ 1889us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:53.584730 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:53.584741 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:53.585282 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:41:53.585811 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:53.585828 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:53.586828 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:53.586834 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:53.588091 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:53.588104 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:53.588812 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:53.588819 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:53.589044 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:53.589629 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:41:53.589650 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:41:53.591712 (+ 2062us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:41:53.594688 (+ 2976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:53.594698 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:53.594714 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:53.595274 (+ 560us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:53.595649 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:53.595654 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:53.595658 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:53.595890 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:53.596406 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:41:53.596413 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:41:53.596810 (+ 397us) client_negotiation.cc:769] Sending connection context
0620 18:41:53.597096 (+ 286us) client_negotiation.cc:240] Negotiation successful
0620 18:41:53.597621 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"thread_start_us":325,"threads_started":1}
I20260620 18:41:53.600478 7254 heartbeater.cc:347] Connected to a master server at 127.23.115.62:44779
I20260620 18:41:53.600762 7254 heartbeater.cc:464] Registering TS with master...
I20260620 18:41:53.601507 7254 heartbeater.cc:511] Master 127.23.115.62:44779 requested a full tablet report, sending...
I20260620 18:41:53.603731 7065 ts_manager.cc:194] Registered new tserver with Master: 86629c1da49b4263ae52078ef6a60192 (127.23.115.1:46605)
I20260620 18:41:53.605207 7065 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.115.1@KRBTEST.COM'} at 127.23.115.1:48115
WARNING: no policy specified for kudu/127.23.115.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.115.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.115.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.2.keytab.
Entry for principal kudu/127.23.115.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.2.keytab.
I20260620 18:41:53.635756 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.115.2:0
--local_ip_for_outbound_sockets=127.23.115.2
--webserver_interface=127.23.115.2
--webserver_port=0
--tserver_master_addrs=127.23.115.62:44779
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.2.keytab
--principal=kudu/127.23.115.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:41715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:41:53.969338 7260 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:41:53.969600 7260 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:41:53.969753 7260 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:41:53.976984 7260 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:41:53.977087 7260 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:41:53.977239 7260 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.115.2
I20260620 18:41:53.986914 7260 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:41715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.2.keytab
--principal=kudu/127.23.115.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.115.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.115.2
--webserver_port=0
--tserver_master_addrs=127.23.115.62:44779
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.115.2
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:41:53.988171 7260 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:41:53.989408 7260 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:41:54.002523 7265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:54.002465 7268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:54.002497 7266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:41:54.003616 7260 server_base.cc:1034] running on GCE node
I20260620 18:41:54.004235 7260 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:41:54.005266 7260 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:41:54.006508 7260 hybrid_clock.cc:648] HybridClock initialized: now 1781980914006456 us; error 73 us; skew 500 ppm
I20260620 18:41:54.007139 7260 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:41:54 dist-test-slave-d645 krb5kdc[7020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980914, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:41:54.010455 7260 init.cc:377] Logged in from keytab as kudu/127.23.115.2@KRBTEST.COM (short username kudu)
I20260620 18:41:54.012001 7260 webserver.cc:460] Webserver started at http://127.23.115.2:37537/ using document root <none> and password file <none>
I20260620 18:41:54.012665 7260 fs_manager.cc:359] Metadata directory not provided
I20260620 18:41:54.012763 7260 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:41:54.013051 7260 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:41:54.016177 7260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-1/data/instance:
uuid: "5de3fa128ee24886be30c46b379343ba"
format_stamp: "Formatted at 2026-06-20 18:41:54 on dist-test-slave-d645"
server_key: "de72182d925d52935bdba1272e6b5600"
server_key_iv: "b67d0c1f4c2e2d24dfdc13b6c80a1dd5"
server_key_version: "encryptionkey@0"
I20260620 18:41:54.016901 7260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-1/wal/instance:
uuid: "5de3fa128ee24886be30c46b379343ba"
format_stamp: "Formatted at 2026-06-20 18:41:54 on dist-test-slave-d645"
server_key: "de72182d925d52935bdba1272e6b5600"
server_key_iv: "b67d0c1f4c2e2d24dfdc13b6c80a1dd5"
server_key_version: "encryptionkey@0"
I20260620 18:41:54.022740 7260 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260620 18:41:54.027000 7275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:54.028707 7260 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 18:41:54.028867 7260 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-1/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-1/wal
uuid: "5de3fa128ee24886be30c46b379343ba"
format_stamp: "Formatted at 2026-06-20 18:41:54 on dist-test-slave-d645"
server_key: "de72182d925d52935bdba1272e6b5600"
server_key_iv: "b67d0c1f4c2e2d24dfdc13b6c80a1dd5"
server_key_version: "encryptionkey@0"
I20260620 18:41:54.029044 7260 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:41:54.062218 7260 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:41:54.066476 7260 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:41:54.066741 7260 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:41:54.067706 7260 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:41:54.069033 7260 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:41:54.069109 7260 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:54.069234 7260 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:41:54.069288 7260 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:54.099133 7260 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.2:35821
I20260620 18:41:54.099174 7387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.2:35821 every 8 connection(s)
I20260620 18:41:54.100330 7260 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-1/data/info.pb
I20260620 18:41:54.108636 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 7260
I20260620 18:41:54.108805 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-1/wal/instance
I20260620 18:41:54.109249 24012 external_mini_cluster.cc:1428] Setting key f4583207b87778b971f18b0d04417c2a
Jun 20 18:41:54 dist-test-slave-d645 krb5kdc[7020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980914, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.2@KRBTEST.COM for kudu/127.23.115.62@KRBTEST.COM
I20260620 18:41:54.120995 7390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:54.102909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.2:38945 (local address 127.23.115.62:44779)
0620 18:41:54.103412 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:54.103418 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:54.104429 (+ 1011us) server_negotiation.cc:407] Connection header received
0620 18:41:54.106297 (+ 1868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:54.106304 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:54.106409 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:54.106556 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:54.108020 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:54.108786 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:54.110537 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:54.111123 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:54.111237 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:54.114887 (+ 3650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:54.114892 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:54.114897 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:54.114964 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:54.117713 (+ 2749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:54.118608 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:54.118612 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:54.118615 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:54.118756 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:54.119267 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:54.119271 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:54.119273 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:54.119551 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:54.119664 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:54.120391 (+ 727us) server_negotiation.cc:299] Negotiation successful
0620 18:41:54.120616 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":192,"threads_started":1}
I20260620 18:41:54.121649 7391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:54.103545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:44779 (local address 127.23.115.2:38945)
0620 18:41:54.104257 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:54.104300 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:54.105950 (+ 1650us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:54.106747 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:54.106756 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:54.107311 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:41:54.107840 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:54.107856 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:54.108951 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:54.108956 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:54.110377 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:54.110391 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:54.111323 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:54.111328 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:54.111561 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:54.112166 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:41:54.112193 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:41:54.114664 (+ 2471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:41:54.117856 (+ 3192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:54.117865 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:54.117882 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:54.118457 (+ 575us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:54.118857 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:54.118862 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:54.118866 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:54.119137 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:54.119662 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:41:54.119668 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:41:54.120000 (+ 332us) client_negotiation.cc:769] Sending connection context
0620 18:41:54.120222 (+ 222us) client_negotiation.cc:240] Negotiation successful
0620 18:41:54.120618 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":299,"threads_started":1}
I20260620 18:41:54.123005 7388 heartbeater.cc:347] Connected to a master server at 127.23.115.62:44779
I20260620 18:41:54.123275 7388 heartbeater.cc:464] Registering TS with master...
I20260620 18:41:54.123989 7388 heartbeater.cc:511] Master 127.23.115.62:44779 requested a full tablet report, sending...
I20260620 18:41:54.125456 7065 ts_manager.cc:194] Registered new tserver with Master: 5de3fa128ee24886be30c46b379343ba (127.23.115.2:35821)
I20260620 18:41:54.126238 7065 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.115.2@KRBTEST.COM'} at 127.23.115.2:38945
WARNING: no policy specified for kudu/127.23.115.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.115.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.115.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.3.keytab.
Entry for principal kudu/127.23.115.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.3.keytab.
I20260620 18:41:54.158141 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.115.3:0
--local_ip_for_outbound_sockets=127.23.115.3
--webserver_interface=127.23.115.3
--webserver_port=0
--tserver_master_addrs=127.23.115.62:44779
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.3.keytab
--principal=kudu/127.23.115.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:41715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:41:54.478948 7394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:41:54.479242 7394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:41:54.479362 7394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:41:54.487010 7394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:41:54.487133 7394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:41:54.487287 7394 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.115.3
I20260620 18:41:54.497133 7394 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:41715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.3.keytab
--principal=kudu/127.23.115.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.115.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.115.3
--webserver_port=0
--tserver_master_addrs=127.23.115.62:44779
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.115.3
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:41:54.498458 7394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:41:54.499707 7394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:41:54.511992 7402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:54.512051 7399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:41:54.511909 7400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:41:54.512959 7394 server_base.cc:1034] running on GCE node
I20260620 18:41:54.513579 7394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:41:54.514508 7394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:41:54.515703 7394 hybrid_clock.cc:648] HybridClock initialized: now 1781980914515657 us; error 54 us; skew 500 ppm
I20260620 18:41:54.516011 7394 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:41:54 dist-test-slave-d645 krb5kdc[7020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980914, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:41:54.518992 7394 init.cc:377] Logged in from keytab as kudu/127.23.115.3@KRBTEST.COM (short username kudu)
I20260620 18:41:54.520516 7394 webserver.cc:460] Webserver started at http://127.23.115.3:34405/ using document root <none> and password file <none>
I20260620 18:41:54.521165 7394 fs_manager.cc:359] Metadata directory not provided
I20260620 18:41:54.521278 7394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:41:54.521550 7394 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:41:54.524521 7394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-2/data/instance:
uuid: "8205483c83dc4727a92ef655e516de62"
format_stamp: "Formatted at 2026-06-20 18:41:54 on dist-test-slave-d645"
server_key: "fb6c68aa3f0f57925a759e5920ef2f3c"
server_key_iv: "417abd0bfa840c397f81c408baafc9c1"
server_key_version: "encryptionkey@0"
I20260620 18:41:54.525274 7394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-2/wal/instance:
uuid: "8205483c83dc4727a92ef655e516de62"
format_stamp: "Formatted at 2026-06-20 18:41:54 on dist-test-slave-d645"
server_key: "fb6c68aa3f0f57925a759e5920ef2f3c"
server_key_iv: "417abd0bfa840c397f81c408baafc9c1"
server_key_version: "encryptionkey@0"
I20260620 18:41:54.530238 7394 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 18:41:54.533897 7409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:54.535322 7394 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 18:41:54.535485 7394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-2/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-2/wal
uuid: "8205483c83dc4727a92ef655e516de62"
format_stamp: "Formatted at 2026-06-20 18:41:54 on dist-test-slave-d645"
server_key: "fb6c68aa3f0f57925a759e5920ef2f3c"
server_key_iv: "417abd0bfa840c397f81c408baafc9c1"
server_key_version: "encryptionkey@0"
I20260620 18:41:54.535682 7394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:41:54.550379 7394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:41:54.554411 7394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:41:54.554666 7394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:41:54.555686 7394 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:41:54.556998 7394 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:41:54.557070 7394 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:54.557183 7394 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:41:54.557235 7394 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:41:54.591158 7394 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.3:38983
I20260620 18:41:54.591223 7521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.3:38983 every 8 connection(s)
I20260620 18:41:54.592392 7394 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-2/data/info.pb
I20260620 18:41:54.600268 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 7394
I20260620 18:41:54.600487 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-2/wal/instance
I20260620 18:41:54.600961 24012 external_mini_cluster.cc:1428] Setting key d146428015257db8705fb4730ac50516
Jun 20 18:41:54 dist-test-slave-d645 krb5kdc[7020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980914, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.3@KRBTEST.COM for kudu/127.23.115.62@KRBTEST.COM
I20260620 18:41:54.608243 7254 heartbeater.cc:503] Master 127.23.115.62:44779 was elected leader, sending a full tablet report...
I20260620 18:41:54.616919 7390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:54.595836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.3:60335 (local address 127.23.115.62:44779)
0620 18:41:54.596003 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:54.596007 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:54.597779 (+ 1772us) server_negotiation.cc:407] Connection header received
0620 18:41:54.600026 (+ 2247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:54.600034 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:54.600158 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:54.600326 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:54.602058 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:54.603144 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:54.604815 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:54.605285 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:54.605401 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:54.609034 (+ 3633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:54.609043 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:54.609048 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:54.609110 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:54.612754 (+ 3644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:54.613879 (+ 1125us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:54.613883 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:54.613886 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:54.614018 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:54.614609 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:54.614616 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:54.614620 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:54.615100 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:54.615294 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:54.616369 (+ 1075us) server_negotiation.cc:299] Negotiation successful
0620 18:41:54.616602 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 18:41:54.617587 7524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:54.596243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:44779 (local address 127.23.115.3:60335)
0620 18:41:54.597478 (+ 1235us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:54.597562 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:54.599651 (+ 2089us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:54.600602 (+ 951us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:54.600611 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:54.601199 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:41:54.601879 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:54.601908 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:54.603297 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:54.603301 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:54.604659 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:54.604672 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:54.605508 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:54.605514 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:54.605743 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:54.606382 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:41:54.606406 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:41:54.608643 (+ 2237us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:41:54.612972 (+ 4329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:54.612983 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:54.613005 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:54.613715 (+ 710us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:54.614179 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:41:54.614184 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:41:54.614187 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:41:54.614449 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:41:54.615304 (+ 855us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:41:54.615313 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:41:54.615710 (+ 397us) client_negotiation.cc:769] Sending connection context
0620 18:41:54.615973 (+ 263us) client_negotiation.cc:240] Negotiation successful
0620 18:41:54.616484 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":745,"thread_start_us":458,"threads_started":1}
I20260620 18:41:54.618876 7522 heartbeater.cc:347] Connected to a master server at 127.23.115.62:44779
I20260620 18:41:54.619190 7522 heartbeater.cc:464] Registering TS with master...
I20260620 18:41:54.619899 7522 heartbeater.cc:511] Master 127.23.115.62:44779 requested a full tablet report, sending...
I20260620 18:41:54.621277 7065 ts_manager.cc:194] Registered new tserver with Master: 8205483c83dc4727a92ef655e516de62 (127.23.115.3:38983)
I20260620 18:41:54.622079 7065 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.115.3@KRBTEST.COM'} at 127.23.115.3:60335
I20260620 18:41:54.628798 24012 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:41:54.649087 7390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:54.633291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33100 (local address 127.23.115.62:44779)
0620 18:41:54.633465 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:54.633471 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:54.634180 (+ 709us) server_negotiation.cc:407] Connection header received
0620 18:41:54.634385 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:54.634389 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:54.634454 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:54.634537 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:54.635909 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:54.636663 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:54.638003 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:54.638731 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:54.638912 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:54.642048 (+ 3136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:54.642055 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:54.642059 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:54.642112 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:54.645751 (+ 3639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:54.646717 (+ 966us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:54.646722 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:54.646725 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:54.646874 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:54.647297 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:54.647301 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:54.647303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:54.647704 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:54.647982 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:54.648347 (+ 365us) server_negotiation.cc:299] Negotiation successful
0620 18:41:54.648622 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 18:41:54.654791 7065 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33100:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:41:54.661612 7065 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:41:54.696141 7536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:54.674087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49674 (local address 127.23.115.1:46605)
0620 18:41:54.687151 (+ 13064us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:54.687157 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:54.687182 (+ 25us) server_negotiation.cc:407] Connection header received
0620 18:41:54.687260 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:54.687265 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:54.687490 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:54.687685 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:41:54.689646 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:54.691173 (+ 1527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:54.693825 (+ 2652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:54.694792 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:54.694914 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:54.695280 (+ 366us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:54.695635 (+ 355us) server_negotiation.cc:299] Negotiation successful
0620 18:41:54.695813 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12920,"thread_start_us":12724,"threads_started":1}
I20260620 18:41:54.696195 7535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:54.673805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.1:46605 (local address 127.0.0.1:49674)
0620 18:41:54.686126 (+ 12321us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:54.686182 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:54.686464 (+ 282us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:54.687945 (+ 1481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:54.687951 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:54.687977 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:41:54.689435 (+ 1458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:54.689456 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:54.691290 (+ 1834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:54.691295 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:54.693679 (+ 2384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:54.693691 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:54.695020 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:54.695027 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:54.695373 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:54.695432 (+ 59us) client_negotiation.cc:769] Sending connection context
0620 18:41:54.695581 (+ 149us) client_negotiation.cc:240] Negotiation successful
0620 18:41:54.695772 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12002,"mutex_wait_us":47,"thread_start_us":15059,"threads_started":1}
I20260620 18:41:54.698262 7538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:54.687114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33478 (local address 127.23.115.3:38983)
0620 18:41:54.687877 (+ 763us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:54.687884 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:54.689992 (+ 2108us) server_negotiation.cc:407] Connection header received
0620 18:41:54.690233 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:54.690239 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:54.690438 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:54.690632 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:41:54.691242 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:54.692741 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:54.695424 (+ 2683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:54.696805 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:54.696926 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:54.697211 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:54.697379 (+ 168us) server_negotiation.cc:299] Negotiation successful
0620 18:41:54.697558 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":449,"threads_started":1}
I20260620 18:41:54.699285 7537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:54.674231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40210 (local address 127.23.115.2:35821)
0620 18:41:54.685223 (+ 10992us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:54.685236 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:54.689572 (+ 4336us) server_negotiation.cc:407] Connection header received
0620 18:41:54.689793 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:54.689801 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:54.690167 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:54.690379 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:41:54.691026 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:54.692572 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:54.695373 (+ 2801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:54.696917 (+ 1544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:54.697064 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:54.697248 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:54.698706 (+ 1458us) server_negotiation.cc:299] Negotiation successful
0620 18:41:54.698885 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10756,"thread_start_us":11015,"threads_started":1}
I20260620 18:41:54.699501 7539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:54.673941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.2:35821 (local address 127.0.0.1:40210)
0620 18:41:54.689425 (+ 15484us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:54.689445 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:54.689588 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:54.690369 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:54.690372 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:54.690395 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:41:54.690814 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:54.690827 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:54.692744 (+ 1917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:54.692749 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:54.695165 (+ 2416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:54.695181 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:54.697369 (+ 2188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:54.697375 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:54.697540 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:54.697569 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 18:41:54.697642 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 18:41:54.697935 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15339,"mutex_wait_us":52,"thread_start_us":11713,"threads_started":1}
I20260620 18:41:54.699693 7534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:54.687906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.3:38983 (local address 127.0.0.1:33478)
0620 18:41:54.689887 (+ 1981us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:54.689903 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:54.690087 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:54.690622 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:54.690627 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:54.690653 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:41:54.691068 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:54.691078 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:54.692895 (+ 1817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:54.692904 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:54.695271 (+ 2367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:54.695283 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:54.696988 (+ 1705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:54.696995 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:54.697162 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:54.697193 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 18:41:54.698641 (+ 1448us) client_negotiation.cc:240] Negotiation successful
0620 18:41:54.698781 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1877,"mutex_wait_us":33,"thread_start_us":1207,"threads_started":1}
I20260620 18:41:54.700989 7189 tablet_service.cc:1467] Processing CreateTablet for tablet 902f740120364f1db6ce2ee5bbfd6514 (DEFAULT_TABLE table=test-table [id=fc0b44324b20434fb160b25656b4c496]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:41:54.702584 7189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 902f740120364f1db6ce2ee5bbfd6514. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:41:54.703150 7457 tablet_service.cc:1467] Processing CreateTablet for tablet 902f740120364f1db6ce2ee5bbfd6514 (DEFAULT_TABLE table=test-table [id=fc0b44324b20434fb160b25656b4c496]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:41:54.703102 7323 tablet_service.cc:1467] Processing CreateTablet for tablet 902f740120364f1db6ce2ee5bbfd6514 (DEFAULT_TABLE table=test-table [id=fc0b44324b20434fb160b25656b4c496]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:41:54.704320 7457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 902f740120364f1db6ce2ee5bbfd6514. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:41:54.704414 7323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 902f740120364f1db6ce2ee5bbfd6514. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:41:54.718477 7540 tablet_bootstrap.cc:492] T 902f740120364f1db6ce2ee5bbfd6514 P 8205483c83dc4727a92ef655e516de62: Bootstrap starting.
I20260620 18:41:54.720940 7542 tablet_bootstrap.cc:492] T 902f740120364f1db6ce2ee5bbfd6514 P 86629c1da49b4263ae52078ef6a60192: Bootstrap starting.
I20260620 18:41:54.722411 7540 tablet_bootstrap.cc:654] T 902f740120364f1db6ce2ee5bbfd6514 P 8205483c83dc4727a92ef655e516de62: Neither blocks nor log segments found. Creating new log.
I20260620 18:41:54.723876 7540 log.cc:824] T 902f740120364f1db6ce2ee5bbfd6514 P 8205483c83dc4727a92ef655e516de62: Log is configured to *not* fsync() on all Append() calls
I20260620 18:41:54.724790 7542 tablet_bootstrap.cc:654] T 902f740120364f1db6ce2ee5bbfd6514 P 86629c1da49b4263ae52078ef6a60192: Neither blocks nor log segments found. Creating new log.
I20260620 18:41:54.725399 7541 tablet_bootstrap.cc:492] T 902f740120364f1db6ce2ee5bbfd6514 P 5de3fa128ee24886be30c46b379343ba: Bootstrap starting.
I20260620 18:41:54.725862 7542 log.cc:824] T 902f740120364f1db6ce2ee5bbfd6514 P 86629c1da49b4263ae52078ef6a60192: Log is configured to *not* fsync() on all Append() calls
I20260620 18:41:54.727324 7540 tablet_bootstrap.cc:492] T 902f740120364f1db6ce2ee5bbfd6514 P 8205483c83dc4727a92ef655e516de62: No bootstrap required, opened a new log
I20260620 18:41:54.727605 7540 ts_tablet_manager.cc:1397] T 902f740120364f1db6ce2ee5bbfd6514 P 8205483c83dc4727a92ef655e516de62: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260620 18:41:54.728382 7541 tablet_bootstrap.cc:654] T 902f740120364f1db6ce2ee5bbfd6514 P 5de3fa128ee24886be30c46b379343ba: Neither blocks nor log segments found. Creating new log.
I20260620 18:41:54.729283 7541 log.cc:824] T 902f740120364f1db6ce2ee5bbfd6514 P 5de3fa128ee24886be30c46b379343ba: Log is configured to *not* fsync() on all Append() calls
I20260620 18:41:54.729679 7542 tablet_bootstrap.cc:492] T 902f740120364f1db6ce2ee5bbfd6514 P 86629c1da49b4263ae52078ef6a60192: No bootstrap required, opened a new log
I20260620 18:41:54.729933 7542 ts_tablet_manager.cc:1397] T 902f740120364f1db6ce2ee5bbfd6514 P 86629c1da49b4263ae52078ef6a60192: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260620 18:41:54.733197 7541 tablet_bootstrap.cc:492] T 902f740120364f1db6ce2ee5bbfd6514 P 5de3fa128ee24886be30c46b379343ba: No bootstrap required, opened a new log
I20260620 18:41:54.733441 7541 ts_tablet_manager.cc:1397] T 902f740120364f1db6ce2ee5bbfd6514 P 5de3fa128ee24886be30c46b379343ba: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20260620 18:41:54.737988 7540 raft_consensus.cc:348] T 902f740120364f1db6ce2ee5bbfd6514 P 8205483c83dc4727a92ef655e516de62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86629c1da49b4263ae52078ef6a60192" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 46605 } } peers { permanent_uuid: "5de3fa128ee24886be30c46b379343ba" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 35821 } } peers { permanent_uuid: "8205483c83dc4727a92ef655e516de62" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 38983 } }
I20260620 18:41:54.738433 7540 raft_consensus.cc:374] T 902f740120364f1db6ce2ee5bbfd6514 P 8205483c83dc4727a92ef655e516de62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:41:54.738544 7540 raft_consensus.cc:729] T 902f740120364f1db6ce2ee5bbfd6514 P 8205483c83dc4727a92ef655e516de62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8205483c83dc4727a92ef655e516de62, State: Initialized, Role: FOLLOWER
I20260620 18:41:54.739334 7540 consensus_queue.cc:260] T 902f740120364f1db6ce2ee5bbfd6514 P 8205483c83dc4727a92ef655e516de62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86629c1da49b4263ae52078ef6a60192" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 46605 } } peers { permanent_uuid: "5de3fa128ee24886be30c46b379343ba" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 35821 } } peers { permanent_uuid: "8205483c83dc4727a92ef655e516de62" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 38983 } }
I20260620 18:41:54.740079 7542 raft_consensus.cc:348] T 902f740120364f1db6ce2ee5bbfd6514 P 86629c1da49b4263ae52078ef6a60192 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86629c1da49b4263ae52078ef6a60192" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 46605 } } peers { permanent_uuid: "5de3fa128ee24886be30c46b379343ba" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 35821 } } peers { permanent_uuid: "8205483c83dc4727a92ef655e516de62" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 38983 } }
I20260620 18:41:54.740538 7542 raft_consensus.cc:374] T 902f740120364f1db6ce2ee5bbfd6514 P 86629c1da49b4263ae52078ef6a60192 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:41:54.740633 7542 raft_consensus.cc:729] T 902f740120364f1db6ce2ee5bbfd6514 P 86629c1da49b4263ae52078ef6a60192 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86629c1da49b4263ae52078ef6a60192, State: Initialized, Role: FOLLOWER
I20260620 18:41:54.741266 7542 consensus_queue.cc:260] T 902f740120364f1db6ce2ee5bbfd6514 P 86629c1da49b4263ae52078ef6a60192 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86629c1da49b4263ae52078ef6a60192" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 46605 } } peers { permanent_uuid: "5de3fa128ee24886be30c46b379343ba" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 35821 } } peers { permanent_uuid: "8205483c83dc4727a92ef655e516de62" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 38983 } }
I20260620 18:41:54.742409 7541 raft_consensus.cc:348] T 902f740120364f1db6ce2ee5bbfd6514 P 5de3fa128ee24886be30c46b379343ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86629c1da49b4263ae52078ef6a60192" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 46605 } } peers { permanent_uuid: "5de3fa128ee24886be30c46b379343ba" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 35821 } } peers { permanent_uuid: "8205483c83dc4727a92ef655e516de62" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 38983 } }
I20260620 18:41:54.742820 7541 raft_consensus.cc:374] T 902f740120364f1db6ce2ee5bbfd6514 P 5de3fa128ee24886be30c46b379343ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:41:54.742908 7541 raft_consensus.cc:729] T 902f740120364f1db6ce2ee5bbfd6514 P 5de3fa128ee24886be30c46b379343ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5de3fa128ee24886be30c46b379343ba, State: Initialized, Role: FOLLOWER
I20260620 18:41:54.743182 7542 ts_tablet_manager.cc:1428] T 902f740120364f1db6ce2ee5bbfd6514 P 86629c1da49b4263ae52078ef6a60192: Time spent starting tablet: real 0.013s user 0.015s sys 0.000s
I20260620 18:41:54.743561 7541 consensus_queue.cc:260] T 902f740120364f1db6ce2ee5bbfd6514 P 5de3fa128ee24886be30c46b379343ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86629c1da49b4263ae52078ef6a60192" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 46605 } } peers { permanent_uuid: "5de3fa128ee24886be30c46b379343ba" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 35821 } } peers { permanent_uuid: "8205483c83dc4727a92ef655e516de62" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 38983 } }
I20260620 18:41:54.745239 7541 ts_tablet_manager.cc:1428] T 902f740120364f1db6ce2ee5bbfd6514 P 5de3fa128ee24886be30c46b379343ba: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260620 18:41:54.746104 7522 heartbeater.cc:503] Master 127.23.115.62:44779 was elected leader, sending a full tablet report...
I20260620 18:41:54.746475 7388 heartbeater.cc:503] Master 127.23.115.62:44779 was elected leader, sending a full tablet report...
I20260620 18:41:54.746575 7540 ts_tablet_manager.cc:1428] T 902f740120364f1db6ce2ee5bbfd6514 P 8205483c83dc4727a92ef655e516de62: Time spent starting tablet: real 0.019s user 0.014s sys 0.004s
W20260620 18:41:54.830721 7255 tablet.cc:2369] T 902f740120364f1db6ce2ee5bbfd6514 P 86629c1da49b4263ae52078ef6a60192: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:41:54.843945 7523 tablet.cc:2369] T 902f740120364f1db6ce2ee5bbfd6514 P 8205483c83dc4727a92ef655e516de62: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:41:54.852304 7389 tablet.cc:2369] T 902f740120364f1db6ce2ee5bbfd6514 P 5de3fa128ee24886be30c46b379343ba: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:41:55.096386 7546 raft_consensus.cc:482] T 902f740120364f1db6ce2ee5bbfd6514 P 8205483c83dc4727a92ef655e516de62 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:41:55.096895 7546 raft_consensus.cc:504] T 902f740120364f1db6ce2ee5bbfd6514 P 8205483c83dc4727a92ef655e516de62 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86629c1da49b4263ae52078ef6a60192" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 46605 } } peers { permanent_uuid: "5de3fa128ee24886be30c46b379343ba" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 35821 } } peers { permanent_uuid: "8205483c83dc4727a92ef655e516de62" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 38983 } }
I20260620 18:41:55.099210 7546 leader_election.cc:288] T 902f740120364f1db6ce2ee5bbfd6514 P 8205483c83dc4727a92ef655e516de62 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 86629c1da49b4263ae52078ef6a60192 (127.23.115.1:46605), 5de3fa128ee24886be30c46b379343ba (127.23.115.2:35821)
I20260620 18:41:55.106289 7537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:55.099730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.3:56087 (local address 127.23.115.2:35821)
0620 18:41:55.099968 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:55.099974 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:55.100076 (+ 102us) server_negotiation.cc:407] Connection header received
0620 18:41:55.100287 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:55.100293 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:55.100380 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:55.100509 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:41:55.101048 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:55.102101 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:55.104498 (+ 2397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:55.105425 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:55.105567 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:55.105641 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:55.105916 (+ 275us) server_negotiation.cc:299] Negotiation successful
0620 18:41:55.106050 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260620 18:41:55.106308 7524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:55.099668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.2:35821 (local address 127.23.115.3:56087)
0620 18:41:55.099969 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:55.099991 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:55.100142 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:55.100683 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:55.100689 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:55.100718 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:41:55.100904 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:55.100914 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:55.102209 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:55.102214 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:55.104300 (+ 2086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:55.104315 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:55.105576 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:55.105581 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:55.105763 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:55.105787 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 18:41:55.105869 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 18:41:55.106002 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"mutex_wait_us":1}
I20260620 18:41:55.107372 7536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:55.099670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.3:41545 (local address 127.23.115.1:46605)
0620 18:41:55.099867 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:55.099873 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:55.100792 (+ 919us) server_negotiation.cc:407] Connection header received
0620 18:41:55.100956 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:55.100960 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:55.101028 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:55.101116 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:41:55.101685 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:55.102669 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:55.104723 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:55.105937 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:55.106079 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:55.106147 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:55.107013 (+ 866us) server_negotiation.cc:299] Negotiation successful
0620 18:41:55.107153 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 18:41:55.107354 7343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "902f740120364f1db6ce2ee5bbfd6514" candidate_uuid: "8205483c83dc4727a92ef655e516de62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5de3fa128ee24886be30c46b379343ba" is_pre_election: true
I20260620 18:41:55.107715 7549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:55.100122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.1:46605 (local address 127.23.115.3:41545)
0620 18:41:55.100692 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0620 18:41:55.100706 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 18:41:55.100820 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:41:55.101139 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:41:55.101144 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:41:55.101169 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:41:55.101515 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:55.101526 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:55.102826 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:55.102830 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:55.104564 (+ 1734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:41:55.104579 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:41:55.106273 (+ 1694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:41:55.106278 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:41:55.106813 (+ 535us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:55.106845 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 18:41:55.106975 (+ 130us) client_negotiation.cc:240] Negotiation successful
0620 18:41:55.107119 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"mutex_wait_us":29,"thread_start_us":324,"threads_started":1}
I20260620 18:41:55.107988 7343 raft_consensus.cc:2454] T 902f740120364f1db6ce2ee5bbfd6514 P 5de3fa128ee24886be30c46b379343ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8205483c83dc4727a92ef655e516de62 in term 0.
I20260620 18:41:55.108752 7209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "902f740120364f1db6ce2ee5bbfd6514" candidate_uuid: "8205483c83dc4727a92ef655e516de62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86629c1da49b4263ae52078ef6a60192" is_pre_election: true
I20260620 18:41:55.109228 7209 raft_consensus.cc:2454] T 902f740120364f1db6ce2ee5bbfd6514 P 86629c1da49b4263ae52078ef6a60192 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8205483c83dc4727a92ef655e516de62 in term 0.
I20260620 18:41:55.109876 7411 leader_election.cc:302] T 902f740120364f1db6ce2ee5bbfd6514 P 8205483c83dc4727a92ef655e516de62 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8205483c83dc4727a92ef655e516de62, 86629c1da49b4263ae52078ef6a60192; no voters:
I20260620 18:41:55.110293 7546 raft_consensus.cc:2781] T 902f740120364f1db6ce2ee5bbfd6514 P 8205483c83dc4727a92ef655e516de62 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:41:55.110424 7546 raft_consensus.cc:482] T 902f740120364f1db6ce2ee5bbfd6514 P 8205483c83dc4727a92ef655e516de62 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:41:55.110522 7546 raft_consensus.cc:3037] T 902f740120364f1db6ce2ee5bbfd6514 P 8205483c83dc4727a92ef655e516de62 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:41:55.112835 7546 raft_consensus.cc:504] T 902f740120364f1db6ce2ee5bbfd6514 P 8205483c83dc4727a92ef655e516de62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86629c1da49b4263ae52078ef6a60192" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 46605 } } peers { permanent_uuid: "5de3fa128ee24886be30c46b379343ba" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 35821 } } peers { permanent_uuid: "8205483c83dc4727a92ef655e516de62" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 38983 } }
I20260620 18:41:55.113525 7546 leader_election.cc:288] T 902f740120364f1db6ce2ee5bbfd6514 P 8205483c83dc4727a92ef655e516de62 [CANDIDATE]: Term 1 election: Requested vote from peers 86629c1da49b4263ae52078ef6a60192 (127.23.115.1:46605), 5de3fa128ee24886be30c46b379343ba (127.23.115.2:35821)
I20260620 18:41:55.113947 7209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "902f740120364f1db6ce2ee5bbfd6514" candidate_uuid: "8205483c83dc4727a92ef655e516de62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86629c1da49b4263ae52078ef6a60192"
I20260620 18:41:55.114051 7343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "902f740120364f1db6ce2ee5bbfd6514" candidate_uuid: "8205483c83dc4727a92ef655e516de62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5de3fa128ee24886be30c46b379343ba"
I20260620 18:41:55.114140 7209 raft_consensus.cc:3037] T 902f740120364f1db6ce2ee5bbfd6514 P 86629c1da49b4263ae52078ef6a60192 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:41:55.114236 7343 raft_consensus.cc:3037] T 902f740120364f1db6ce2ee5bbfd6514 P 5de3fa128ee24886be30c46b379343ba [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:41:55.116359 7209 raft_consensus.cc:2454] T 902f740120364f1db6ce2ee5bbfd6514 P 86629c1da49b4263ae52078ef6a60192 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8205483c83dc4727a92ef655e516de62 in term 1.
I20260620 18:41:55.116709 7411 leader_election.cc:302] T 902f740120364f1db6ce2ee5bbfd6514 P 8205483c83dc4727a92ef655e516de62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8205483c83dc4727a92ef655e516de62, 86629c1da49b4263ae52078ef6a60192; no voters:
I20260620 18:41:55.116958 7546 raft_consensus.cc:2781] T 902f740120364f1db6ce2ee5bbfd6514 P 8205483c83dc4727a92ef655e516de62 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:41:55.117038 7343 raft_consensus.cc:2454] T 902f740120364f1db6ce2ee5bbfd6514 P 5de3fa128ee24886be30c46b379343ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8205483c83dc4727a92ef655e516de62 in term 1.
I20260620 18:41:55.117416 7546 raft_consensus.cc:686] T 902f740120364f1db6ce2ee5bbfd6514 P 8205483c83dc4727a92ef655e516de62 [term 1 LEADER]: Becoming Leader. State: Replica: 8205483c83dc4727a92ef655e516de62, State: Running, Role: LEADER
I20260620 18:41:55.117899 7546 consensus_queue.cc:237] T 902f740120364f1db6ce2ee5bbfd6514 P 8205483c83dc4727a92ef655e516de62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86629c1da49b4263ae52078ef6a60192" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 46605 } } peers { permanent_uuid: "5de3fa128ee24886be30c46b379343ba" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 35821 } } peers { permanent_uuid: "8205483c83dc4727a92ef655e516de62" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 38983 } }
I20260620 18:41:55.122608 7065 catalog_manager.cc:5495] T 902f740120364f1db6ce2ee5bbfd6514 P 8205483c83dc4727a92ef655e516de62 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8205483c83dc4727a92ef655e516de62 (127.23.115.3). New cstate: current_term: 1 leader_uuid: "8205483c83dc4727a92ef655e516de62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86629c1da49b4263ae52078ef6a60192" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 46605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5de3fa128ee24886be30c46b379343ba" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 35821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8205483c83dc4727a92ef655e516de62" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 38983 } health_report { overall_health: HEALTHY } } }
I20260620 18:41:55.171020 7538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:55.164358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33492 (local address 127.23.115.3:38983)
0620 18:41:55.164627 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:55.164633 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:55.164984 (+ 351us) server_negotiation.cc:407] Connection header received
0620 18:41:55.165169 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:55.165173 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:55.165305 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:55.165450 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:41:55.166034 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:55.166854 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:55.168518 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:55.168912 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:55.169027 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:55.169874 (+ 847us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:41:55.170434 (+ 560us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:41:55.170550 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:55.170670 (+ 120us) server_negotiation.cc:299] Negotiation successful
0620 18:41:55.170741 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260620 18:41:55.178736 7209 raft_consensus.cc:1261] T 902f740120364f1db6ce2ee5bbfd6514 P 86629c1da49b4263ae52078ef6a60192 [term 1 FOLLOWER]: Refusing update from remote peer 8205483c83dc4727a92ef655e516de62: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:41:55.178735 7343 raft_consensus.cc:1261] T 902f740120364f1db6ce2ee5bbfd6514 P 5de3fa128ee24886be30c46b379343ba [term 1 FOLLOWER]: Refusing update from remote peer 8205483c83dc4727a92ef655e516de62: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:41:55.179693 7550 consensus_queue.cc:1035] T 902f740120364f1db6ce2ee5bbfd6514 P 8205483c83dc4727a92ef655e516de62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86629c1da49b4263ae52078ef6a60192" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 46605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:41:55.180042 7546 consensus_queue.cc:1035] T 902f740120364f1db6ce2ee5bbfd6514 P 8205483c83dc4727a92ef655e516de62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5de3fa128ee24886be30c46b379343ba" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 35821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:41:55.194557 7556 mvcc.cc:205] Tried to move back new op lower bound from 7298993828563103744 to 7298993828326346752. Current Snapshot: MvccSnapshot[applied={T|T < 7298993828563103744}]
I20260620 18:41:55.206804 7063 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33100:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:41:55.207130 7063 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33100:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:41:55.210414 7063 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2ea2131d7e994f00a8a2e0f24724e171: Sending DeleteTablet for 3 replicas of tablet 902f740120364f1db6ce2ee5bbfd6514
I20260620 18:41:55.211732 7323 tablet_service.cc:1514] Processing DeleteTablet for tablet 902f740120364f1db6ce2ee5bbfd6514 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:41:55 UTC) from {username='kudu', principal='kudu/127.23.115.62@KRBTEST.COM'} at 127.0.0.1:40210
I20260620 18:41:55.211720 7457 tablet_service.cc:1514] Processing DeleteTablet for tablet 902f740120364f1db6ce2ee5bbfd6514 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:41:55 UTC) from {username='kudu', principal='kudu/127.23.115.62@KRBTEST.COM'} at 127.0.0.1:33478
I20260620 18:41:55.211721 7189 tablet_service.cc:1514] Processing DeleteTablet for tablet 902f740120364f1db6ce2ee5bbfd6514 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:41:55 UTC) from {username='kudu', principal='kudu/127.23.115.62@KRBTEST.COM'} at 127.0.0.1:49674
I20260620 18:41:55.212172 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 7126
I20260620 18:41:55.222654 7566 tablet_replica.cc:330] T 902f740120364f1db6ce2ee5bbfd6514 P 86629c1da49b4263ae52078ef6a60192: stopping tablet replica
I20260620 18:41:55.223251 7566 raft_consensus.cc:2229] T 902f740120364f1db6ce2ee5bbfd6514 P 86629c1da49b4263ae52078ef6a60192 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:41:55.224363 7564 tablet_replica.cc:330] T 902f740120364f1db6ce2ee5bbfd6514 P 5de3fa128ee24886be30c46b379343ba: stopping tablet replica
I20260620 18:41:55.225092 7564 raft_consensus.cc:2229] T 902f740120364f1db6ce2ee5bbfd6514 P 5de3fa128ee24886be30c46b379343ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:41:55.226058 7566 raft_consensus.cc:2258] T 902f740120364f1db6ce2ee5bbfd6514 P 86629c1da49b4263ae52078ef6a60192 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:41:55.226786 7564 raft_consensus.cc:2258] T 902f740120364f1db6ce2ee5bbfd6514 P 5de3fa128ee24886be30c46b379343ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:41:55.228684 7566 ts_tablet_manager.cc:1905] T 902f740120364f1db6ce2ee5bbfd6514 P 86629c1da49b4263ae52078ef6a60192: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:41:55.231259 7565 tablet_replica.cc:330] T 902f740120364f1db6ce2ee5bbfd6514 P 8205483c83dc4727a92ef655e516de62: stopping tablet replica
I20260620 18:41:55.233094 7564 ts_tablet_manager.cc:1905] T 902f740120364f1db6ce2ee5bbfd6514 P 5de3fa128ee24886be30c46b379343ba: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 20 18:41:55 dist-test-slave-d645 krb5kdc[7020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980912, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.115.1@KRBTEST.COM
I20260620 18:41:55.236948 7566 ts_tablet_manager.cc:1918] T 902f740120364f1db6ce2ee5bbfd6514 P 86629c1da49b4263ae52078ef6a60192: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:41:55.237125 7566 log.cc:1196] T 902f740120364f1db6ce2ee5bbfd6514 P 86629c1da49b4263ae52078ef6a60192: Deleting WAL directory at /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-0/wal/wals/902f740120364f1db6ce2ee5bbfd6514
I20260620 18:41:55.238245 7565 raft_consensus.cc:2229] T 902f740120364f1db6ce2ee5bbfd6514 P 8205483c83dc4727a92ef655e516de62 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:41:55.239048 7566 ts_tablet_manager.cc:1939] T 902f740120364f1db6ce2ee5bbfd6514 P 86629c1da49b4263ae52078ef6a60192: Deleting consensus metadata
I20260620 18:41:55.239048 7565 raft_consensus.cc:2258] T 902f740120364f1db6ce2ee5bbfd6514 P 8205483c83dc4727a92ef655e516de62 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:41:55.240127 7564 ts_tablet_manager.cc:1918] T 902f740120364f1db6ce2ee5bbfd6514 P 5de3fa128ee24886be30c46b379343ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:41:55.240286 7564 log.cc:1196] T 902f740120364f1db6ce2ee5bbfd6514 P 5de3fa128ee24886be30c46b379343ba: Deleting WAL directory at /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-1/wal/wals/902f740120364f1db6ce2ee5bbfd6514
I20260620 18:41:55.240754 7564 ts_tablet_manager.cc:1939] T 902f740120364f1db6ce2ee5bbfd6514 P 5de3fa128ee24886be30c46b379343ba: Deleting consensus metadata
I20260620 18:41:55.241128 7565 ts_tablet_manager.cc:1905] T 902f740120364f1db6ce2ee5bbfd6514 P 8205483c83dc4727a92ef655e516de62: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:41:55.242815 7052 catalog_manager.cc:4841] TS 5de3fa128ee24886be30c46b379343ba (127.23.115.2:35821): tablet 902f740120364f1db6ce2ee5bbfd6514 (table test-table [id=fc0b44324b20434fb160b25656b4c496]) successfully deleted
I20260620 18:41:55.245143 7536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:55.213608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49688 (local address 127.23.115.1:46605)
0620 18:41:55.213771 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:55.213778 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:55.222791 (+ 9013us) server_negotiation.cc:407] Connection header received
0620 18:41:55.223386 (+ 595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:55.223394 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:55.223524 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:55.223602 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:55.226359 (+ 2757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:55.227617 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:55.229993 (+ 2376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:55.230676 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:55.230800 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:55.236985 (+ 6185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:55.236999 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:55.237016 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:55.237067 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:55.240761 (+ 3694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:55.241719 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:55.241727 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:55.241731 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:55.241900 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:55.242343 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:55.242348 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:55.242350 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:55.243023 (+ 673us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:55.243310 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:55.244513 (+ 1203us) server_negotiation.cc:299] Negotiation successful
0620 18:41:55.244735 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 18:41:55.245115 7050 catalog_manager.cc:4841] TS 86629c1da49b4263ae52078ef6a60192 (127.23.115.1:46605): tablet 902f740120364f1db6ce2ee5bbfd6514 (table test-table [id=fc0b44324b20434fb160b25656b4c496]) successfully deleted
I20260620 18:41:55.245855 7249 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:41:55.249704 7565 ts_tablet_manager.cc:1918] T 902f740120364f1db6ce2ee5bbfd6514 P 8205483c83dc4727a92ef655e516de62: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:41:55.249887 7565 log.cc:1196] T 902f740120364f1db6ce2ee5bbfd6514 P 8205483c83dc4727a92ef655e516de62: Deleting WAL directory at /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-2/wal/wals/902f740120364f1db6ce2ee5bbfd6514
I20260620 18:41:55.250375 7565 ts_tablet_manager.cc:1939] T 902f740120364f1db6ce2ee5bbfd6514 P 8205483c83dc4727a92ef655e516de62: Deleting consensus metadata
I20260620 18:41:55.252169 7052 catalog_manager.cc:4841] TS 8205483c83dc4727a92ef655e516de62 (127.23.115.3:38983): tablet 902f740120364f1db6ce2ee5bbfd6514 (table test-table [id=fc0b44324b20434fb160b25656b4c496]) successfully deleted
I20260620 18:41:55.336741 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 7126
I20260620 18:41:55.351843 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 7260
Jun 20 18:41:55 dist-test-slave-d645 krb5kdc[7020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980912, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.115.2@KRBTEST.COM
I20260620 18:41:55.369681 7537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:55.352776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40224 (local address 127.23.115.2:35821)
0620 18:41:55.352989 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:55.352996 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:55.353018 (+ 22us) server_negotiation.cc:407] Connection header received
0620 18:41:55.353084 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:55.353089 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:55.353180 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:55.353292 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:55.354820 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:55.355971 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:55.357552 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:55.358206 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:55.358328 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:55.362756 (+ 4428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:55.362777 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:55.362799 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:55.362847 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:55.366571 (+ 3724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:55.367532 (+ 961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:55.367542 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:55.367546 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:55.367710 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:55.368083 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:55.368087 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:55.368091 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:55.368575 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:55.368858 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:55.369132 (+ 274us) server_negotiation.cc:299] Negotiation successful
0620 18:41:55.369350 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 18:41:55.370376 7383 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:41:55.507440 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 7260
I20260620 18:41:55.523192 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 7394
Jun 20 18:41:55 dist-test-slave-d645 krb5kdc[7020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980912, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.115.3@KRBTEST.COM
I20260620 18:41:55.545312 7538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:41:55.524371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33500 (local address 127.23.115.3:38983)
0620 18:41:55.524569 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 18:41:55.524576 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:41:55.524616 (+ 40us) server_negotiation.cc:407] Connection header received
0620 18:41:55.524763 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:41:55.524767 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:41:55.524849 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:41:55.524936 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:41:55.526795 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:55.528125 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:55.530314 (+ 2189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:41:55.530893 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:41:55.531053 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:41:55.537442 (+ 6389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:41:55.537455 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:41:55.537471 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:41:55.537510 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:41:55.541223 (+ 3713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:55.542347 (+ 1124us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:55.542355 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:55.542358 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:55.542661 (+ 303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:41:55.543238 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:41:55.543243 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:41:55.543246 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:41:55.543691 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:41:55.543917 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0620 18:41:55.544534 (+ 617us) server_negotiation.cc:299] Negotiation successful
0620 18:41:55.544895 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 18:41:55.546036 7517 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:41:55.671569 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 7394
I20260620 18:41:55.688886 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 7033
I20260620 18:41:55.690042 7095 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:41:55.800968 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 7033
I20260620 18:42:15.819419 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.115.62:44779
--webserver_interface=127.23.115.62
--webserver_port=36309
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab
--principal=kudu/127.23.115.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:41715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.115.62:44779
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:42:16.163060 7572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:42:16.163340 7572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:42:16.163429 7572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:42:16.170899 7572 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 18:42:16.171026 7572 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 18:42:16.171066 7572 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 18:42:16.171113 7572 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:42:16.171166 7572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:42:16.171232 7572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:42:16.171276 7572 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:42:16.171329 7572 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:42:16.181146 7572 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:41715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.23.115.62:44779
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab
--principal=kudu/127.23.115.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.115.62:44779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.115.62
--webserver_port=36309
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:42:16.182356 7572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:42:16.183688 7572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:42:16.204480 7579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:16.204517 7581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:16.204480 7578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:42:16.205439 7572 server_base.cc:1034] running on GCE node
I20260620 18:42:16.206243 7572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:42:16.207782 7572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:42:16.209008 7572 hybrid_clock.cc:648] HybridClock initialized: now 1781980936208935 us; error 40 us; skew 500 ppm
I20260620 18:42:16.209379 7572 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:42:16 dist-test-slave-d645 krb5kdc[7020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980936, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:42:16.212918 7572 init.cc:377] Logged in from keytab as kudu/127.23.115.62@KRBTEST.COM (short username kudu)
I20260620 18:42:16.214568 7572 webserver.cc:460] Webserver started at http://127.23.115.62:36309/ using document root <none> and password file <none>
I20260620 18:42:16.215339 7572 fs_manager.cc:359] Metadata directory not provided
I20260620 18:42:16.215447 7572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:42:16.221207 7572 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260620 18:42:16.224162 7588 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:16.225485 7572 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 18:42:16.225643 7572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/master-0/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/master-0/wal
uuid: "2ea2131d7e994f00a8a2e0f24724e171"
format_stamp: "Formatted at 2026-06-20 18:41:52 on dist-test-slave-d645"
server_key: "d89fe0e68734db40a03bc14e40aaf1c6"
server_key_iv: "8b96b28491fa7104a042e84a15200d10"
server_key_version: "encryptionkey@0"
I20260620 18:42:16.226720 7572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:42:16.245064 7572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:42:16.249677 7572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:42:16.249936 7572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:42:16.269781 7572 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.62:44779
I20260620 18:42:16.269821 7639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.62:44779 every 8 connection(s)
I20260620 18:42:16.271355 7572 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
I20260620 18:42:16.274439 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 7572
I20260620 18:42:16.275501 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.115.1:46605
--local_ip_for_outbound_sockets=127.23.115.1
--tserver_master_addrs=127.23.115.62:44779
--webserver_port=35971
--webserver_interface=127.23.115.1
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.1.keytab
--principal=kudu/127.23.115.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:41715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:42:16.281186 7640 sys_catalog.cc:263] Verifying existing consensus state
I20260620 18:42:16.285631 7640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ea2131d7e994f00a8a2e0f24724e171: Bootstrap starting.
I20260620 18:42:16.306267 7640 log.cc:824] T 00000000000000000000000000000000 P 2ea2131d7e994f00a8a2e0f24724e171: Log is configured to *not* fsync() on all Append() calls
I20260620 18:42:16.318225 7640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ea2131d7e994f00a8a2e0f24724e171: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 18:42:16.318773 7640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ea2131d7e994f00a8a2e0f24724e171: Bootstrap complete.
I20260620 18:42:16.328289 7640 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ea2131d7e994f00a8a2e0f24724e171 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ea2131d7e994f00a8a2e0f24724e171" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 44779 } }
I20260620 18:42:16.329207 7640 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ea2131d7e994f00a8a2e0f24724e171 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ea2131d7e994f00a8a2e0f24724e171, State: Initialized, Role: FOLLOWER
I20260620 18:42:16.329839 7640 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ea2131d7e994f00a8a2e0f24724e171 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ea2131d7e994f00a8a2e0f24724e171" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 44779 } }
I20260620 18:42:16.330058 7640 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2ea2131d7e994f00a8a2e0f24724e171 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:42:16.330174 7640 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ea2131d7e994f00a8a2e0f24724e171 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:42:16.330336 7640 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ea2131d7e994f00a8a2e0f24724e171 [term 1 FOLLOWER]: Advancing to term 2
I20260620 18:42:16.334517 7640 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ea2131d7e994f00a8a2e0f24724e171 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ea2131d7e994f00a8a2e0f24724e171" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 44779 } }
I20260620 18:42:16.335100 7640 leader_election.cc:302] T 00000000000000000000000000000000 P 2ea2131d7e994f00a8a2e0f24724e171 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ea2131d7e994f00a8a2e0f24724e171; no voters:
I20260620 18:42:16.335683 7640 leader_election.cc:288] T 00000000000000000000000000000000 P 2ea2131d7e994f00a8a2e0f24724e171 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 18:42:16.335822 7644 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2ea2131d7e994f00a8a2e0f24724e171 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 18:42:16.337903 7644 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2ea2131d7e994f00a8a2e0f24724e171 [term 2 LEADER]: Becoming Leader. State: Replica: 2ea2131d7e994f00a8a2e0f24724e171, State: Running, Role: LEADER
I20260620 18:42:16.338507 7644 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ea2131d7e994f00a8a2e0f24724e171 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ea2131d7e994f00a8a2e0f24724e171" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 44779 } }
I20260620 18:42:16.338897 7640 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ea2131d7e994f00a8a2e0f24724e171 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:42:16.341097 7646 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ea2131d7e994f00a8a2e0f24724e171 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ea2131d7e994f00a8a2e0f24724e171. Latest consensus state: current_term: 2 leader_uuid: "2ea2131d7e994f00a8a2e0f24724e171" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ea2131d7e994f00a8a2e0f24724e171" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 44779 } } }
I20260620 18:42:16.341135 7645 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ea2131d7e994f00a8a2e0f24724e171 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "2ea2131d7e994f00a8a2e0f24724e171" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ea2131d7e994f00a8a2e0f24724e171" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 44779 } } }
I20260620 18:42:16.341466 7646 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ea2131d7e994f00a8a2e0f24724e171 [sys.catalog]: This master's current role is: LEADER
I20260620 18:42:16.341506 7645 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ea2131d7e994f00a8a2e0f24724e171 [sys.catalog]: This master's current role is: LEADER
I20260620 18:42:16.352190 7651 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:42:16.360644 7651 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:42:16.362149 7651 catalog_manager.cc:1259] Loaded cluster ID: a10c99f42da74d3bbf169911bfe9f448
I20260620 18:42:16.362253 7651 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:42:16.366608 7651 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:42:16.368245 7651 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2ea2131d7e994f00a8a2e0f24724e171: Loaded TSK: 0
I20260620 18:42:16.405766 7651 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2ea2131d7e994f00a8a2e0f24724e171: Generated new TSK 1
I20260620 18:42:16.406162 7651 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 18:42:16.626304 7642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:42:16.626590 7642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:42:16.626699 7642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:42:16.634367 7642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:42:16.634485 7642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:42:16.634635 7642 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.115.1
I20260620 18:42:16.644737 7642 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:41715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.1.keytab
--principal=kudu/127.23.115.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.115.1:46605
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.115.1
--webserver_port=35971
--tserver_master_addrs=127.23.115.62:44779
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.115.1
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:42:16.646071 7642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:42:16.647343 7642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:42:16.661823 7667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:16.662634 7670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:16.662709 7668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:42:16.663028 7642 server_base.cc:1034] running on GCE node
I20260620 18:42:16.663748 7642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:42:16.664759 7642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:42:16.666009 7642 hybrid_clock.cc:648] HybridClock initialized: now 1781980936665927 us; error 68 us; skew 500 ppm
I20260620 18:42:16.666349 7642 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:42:16 dist-test-slave-d645 krb5kdc[7020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980936, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:42:16.669903 7642 init.cc:377] Logged in from keytab as kudu/127.23.115.1@KRBTEST.COM (short username kudu)
I20260620 18:42:16.671708 7642 webserver.cc:460] Webserver started at http://127.23.115.1:35971/ using document root <none> and password file <none>
I20260620 18:42:16.672407 7642 fs_manager.cc:359] Metadata directory not provided
I20260620 18:42:16.672539 7642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:42:16.678539 7642 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 18:42:16.682324 7677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:16.683804 7642 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:42:16.683969 7642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-0/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-0/wal
uuid: "86629c1da49b4263ae52078ef6a60192"
format_stamp: "Formatted at 2026-06-20 18:41:53 on dist-test-slave-d645"
server_key: "a45a2eff8f0bc3ba6e9dc818944ad5ae"
server_key_iv: "7ae3c4659b337945e16c951f830ad5b9"
server_key_version: "encryptionkey@0"
I20260620 18:42:16.684939 7642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:42:16.706059 7642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:42:16.712119 7642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:42:16.712374 7642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:42:16.713673 7642 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:42:16.715142 7642 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:42:16.715240 7642 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:16.715349 7642 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:42:16.715422 7642 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:16.750963 7642 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.1:46605
I20260620 18:42:16.751039 7789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.1:46605 every 8 connection(s)
I20260620 18:42:16.752403 7642 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-0/data/info.pb
I20260620 18:42:16.762564 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 7642
I20260620 18:42:16.763374 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.115.2:35821
--local_ip_for_outbound_sockets=127.23.115.2
--tserver_master_addrs=127.23.115.62:44779
--webserver_port=37537
--webserver_interface=127.23.115.2
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.2.keytab
--principal=kudu/127.23.115.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:41715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 18:42:16 dist-test-slave-d645 krb5kdc[7020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980936, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.1@KRBTEST.COM for kudu/127.23.115.62@KRBTEST.COM
I20260620 18:42:16.779052 7793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:16.755933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:44779 (local address 127.23.115.1:46627)
0620 18:42:16.756731 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:16.756781 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:16.758491 (+ 1710us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:16.760316 (+ 1825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:16.760324 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:16.760900 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:42:16.761404 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:16.761421 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:16.763235 (+ 1814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:16.763241 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:16.765063 (+ 1822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:16.765081 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:16.766169 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:16.766175 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:16.766420 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:16.767133 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:42:16.767177 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:42:16.769790 (+ 2613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:42:16.774828 (+ 5038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:16.774836 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:16.774849 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:16.775353 (+ 504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:16.775870 (+ 517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:16.775874 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:16.775877 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:16.776061 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:16.776985 (+ 924us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:42:16.776992 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:42:16.777311 (+ 319us) client_negotiation.cc:769] Sending connection context
0620 18:42:16.777558 (+ 247us) client_negotiation.cc:240] Negotiation successful
0620 18:42:16.777994 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":321,"threads_started":1}
I20260620 18:42:16.779312 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:16.755807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.1:46627 (local address 127.23.115.62:44779)
0620 18:42:16.756384 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:16.756396 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:16.756937 (+ 541us) server_negotiation.cc:407] Connection header received
0620 18:42:16.759408 (+ 2471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:16.759430 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:16.759814 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:16.760106 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:16.761618 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:16.763019 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:16.765256 (+ 2237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:16.765923 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:16.766065 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:16.770321 (+ 4256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:16.770333 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:16.770356 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:16.770423 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:16.774654 (+ 4231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:16.775488 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:16.775496 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:16.775503 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:16.775716 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:16.776189 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:16.776193 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:16.776196 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:16.776806 (+ 610us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:16.777115 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:16.777798 (+ 683us) server_negotiation.cc:299] Negotiation successful
0620 18:42:16.778211 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":249,"threads_started":1}
I20260620 18:42:16.782217 7790 heartbeater.cc:347] Connected to a master server at 127.23.115.62:44779
I20260620 18:42:16.782526 7790 heartbeater.cc:464] Registering TS with master...
I20260620 18:42:16.783417 7790 heartbeater.cc:511] Master 127.23.115.62:44779 requested a full tablet report, sending...
I20260620 18:42:16.786432 7605 ts_manager.cc:194] Registered new tserver with Master: 86629c1da49b4263ae52078ef6a60192 (127.23.115.1:46605)
I20260620 18:42:16.788666 7605 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.115.1@KRBTEST.COM'} at 127.23.115.1:46627
W20260620 18:42:17.087973 7794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:42:17.088280 7794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:42:17.088402 7794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:42:17.095718 7794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:42:17.095826 7794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:42:17.095968 7794 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.115.2
I20260620 18:42:17.105553 7794 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:41715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.2.keytab
--principal=kudu/127.23.115.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.115.2:35821
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.115.2
--webserver_port=37537
--tserver_master_addrs=127.23.115.62:44779
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.115.2
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:42:17.106827 7794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:42:17.108065 7794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:42:17.120370 7799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:17.120332 7802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:17.120301 7800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:42:17.121155 7794 server_base.cc:1034] running on GCE node
I20260620 18:42:17.121730 7794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:42:17.122656 7794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:42:17.123883 7794 hybrid_clock.cc:648] HybridClock initialized: now 1781980937123810 us; error 61 us; skew 500 ppm
I20260620 18:42:17.124289 7794 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:42:17 dist-test-slave-d645 krb5kdc[7020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980937, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:42:17.127368 7794 init.cc:377] Logged in from keytab as kudu/127.23.115.2@KRBTEST.COM (short username kudu)
I20260620 18:42:17.128749 7794 webserver.cc:460] Webserver started at http://127.23.115.2:37537/ using document root <none> and password file <none>
I20260620 18:42:17.129396 7794 fs_manager.cc:359] Metadata directory not provided
I20260620 18:42:17.129534 7794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:42:17.134984 7794 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 18:42:17.137660 7809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:17.139072 7794 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:42:17.139200 7794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-1/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-1/wal
uuid: "5de3fa128ee24886be30c46b379343ba"
format_stamp: "Formatted at 2026-06-20 18:41:54 on dist-test-slave-d645"
server_key: "de72182d925d52935bdba1272e6b5600"
server_key_iv: "b67d0c1f4c2e2d24dfdc13b6c80a1dd5"
server_key_version: "encryptionkey@0"
I20260620 18:42:17.140390 7794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:42:17.168565 7794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:42:17.172784 7794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:42:17.173014 7794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:42:17.174008 7794 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:42:17.175433 7794 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:42:17.175522 7794 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:17.175602 7794 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:42:17.175657 7794 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:17.205557 7794 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.2:35821
I20260620 18:42:17.205674 7921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.2:35821 every 8 connection(s)
I20260620 18:42:17.207052 7794 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-1/data/info.pb
I20260620 18:42:17.211505 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 7794
I20260620 18:42:17.212314 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.115.3:38983
--local_ip_for_outbound_sockets=127.23.115.3
--tserver_master_addrs=127.23.115.62:44779
--webserver_port=34405
--webserver_interface=127.23.115.3
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.3.keytab
--principal=kudu/127.23.115.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:41715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 18:42:17 dist-test-slave-d645 krb5kdc[7020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980937, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.2@KRBTEST.COM for kudu/127.23.115.62@KRBTEST.COM
I20260620 18:42:17.230564 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:17.209728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.2:55781 (local address 127.23.115.62:44779)
0620 18:42:17.209935 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:17.209940 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:17.211286 (+ 1346us) server_negotiation.cc:407] Connection header received
0620 18:42:17.213982 (+ 2696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:17.213987 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:17.214063 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:17.214161 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:17.216390 (+ 2229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:17.217253 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:17.219676 (+ 2423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:17.220140 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:17.220280 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:17.224420 (+ 4140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:17.224425 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:17.224429 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:17.224465 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:17.227068 (+ 2603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:17.227930 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:17.227944 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:17.227947 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:17.228091 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:17.228573 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:17.228577 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:17.228579 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:17.228880 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:17.228998 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:17.229902 (+ 904us) server_negotiation.cc:299] Negotiation successful
0620 18:42:17.230159 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260620 18:42:17.231813 7924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:17.210184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:44779 (local address 127.23.115.2:55781)
0620 18:42:17.211096 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:17.211147 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:17.213640 (+ 2493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:17.214456 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:17.214477 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:17.215238 (+ 761us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:42:17.216028 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:17.216053 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:17.217503 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:17.217510 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:17.219445 (+ 1935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:17.219462 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:17.220399 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:17.220407 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:17.220624 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:17.221405 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:42:17.221436 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:42:17.224081 (+ 2645us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:42:17.227202 (+ 3121us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:17.227211 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:17.227230 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:17.227775 (+ 545us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:17.228194 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:17.228199 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:17.228202 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:17.228430 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:17.229005 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:42:17.229014 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:42:17.229460 (+ 446us) client_negotiation.cc:769] Sending connection context
0620 18:42:17.229753 (+ 293us) client_negotiation.cc:240] Negotiation successful
0620 18:42:17.230325 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":351,"threads_started":1}
I20260620 18:42:17.233428 7922 heartbeater.cc:347] Connected to a master server at 127.23.115.62:44779
I20260620 18:42:17.233773 7922 heartbeater.cc:464] Registering TS with master...
I20260620 18:42:17.234460 7922 heartbeater.cc:511] Master 127.23.115.62:44779 requested a full tablet report, sending...
I20260620 18:42:17.236069 7605 ts_manager.cc:194] Registered new tserver with Master: 5de3fa128ee24886be30c46b379343ba (127.23.115.2:35821)
I20260620 18:42:17.236893 7605 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.115.2@KRBTEST.COM'} at 127.23.115.2:55781
W20260620 18:42:17.539258 7925 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:42:17.539573 7925 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:42:17.539710 7925 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:42:17.547469 7925 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:42:17.547592 7925 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:42:17.547744 7925 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.115.3
I20260620 18:42:17.557567 7925 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:41715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.3.keytab
--principal=kudu/127.23.115.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.115.3:38983
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.115.3
--webserver_port=34405
--tserver_master_addrs=127.23.115.62:44779
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.115.3
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:42:17.558780 7925 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:42:17.560047 7925 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:42:17.572239 7930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:17.572260 7931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:17.572338 7933 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:42:17.573048 7925 server_base.cc:1034] running on GCE node
I20260620 18:42:17.573680 7925 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:42:17.574730 7925 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:42:17.575948 7925 hybrid_clock.cc:648] HybridClock initialized: now 1781980937575907 us; error 46 us; skew 500 ppm
I20260620 18:42:17.576256 7925 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:42:17 dist-test-slave-d645 krb5kdc[7020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980937, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:42:17.579330 7925 init.cc:377] Logged in from keytab as kudu/127.23.115.3@KRBTEST.COM (short username kudu)
I20260620 18:42:17.581058 7925 webserver.cc:460] Webserver started at http://127.23.115.3:34405/ using document root <none> and password file <none>
I20260620 18:42:17.581704 7925 fs_manager.cc:359] Metadata directory not provided
I20260620 18:42:17.581826 7925 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:42:17.587594 7925 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 18:42:17.590312 7940 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:17.591655 7925 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:42:17.591804 7925 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-2/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-2/wal
uuid: "8205483c83dc4727a92ef655e516de62"
format_stamp: "Formatted at 2026-06-20 18:41:54 on dist-test-slave-d645"
server_key: "fb6c68aa3f0f57925a759e5920ef2f3c"
server_key_iv: "417abd0bfa840c397f81c408baafc9c1"
server_key_version: "encryptionkey@0"
I20260620 18:42:17.592747 7925 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:42:17.607820 7925 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:42:17.611877 7925 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:42:17.612119 7925 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:42:17.613225 7925 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:42:17.614583 7925 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:42:17.614679 7925 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:17.614787 7925 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:42:17.614858 7925 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:17.648051 7925 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.3:38983
I20260620 18:42:17.648120 8052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.3:38983 every 8 connection(s)
I20260620 18:42:17.649448 7925 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-2/data/info.pb
I20260620 18:42:17.658761 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 7925
Jun 20 18:42:17 dist-test-slave-d645 krb5kdc[7020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980937, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.3@KRBTEST.COM for kudu/127.23.115.62@KRBTEST.COM
I20260620 18:42:17.669868 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:17.651850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.3:45845 (local address 127.23.115.62:44779)
0620 18:42:17.652033 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:17.652037 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:17.653072 (+ 1035us) server_negotiation.cc:407] Connection header received
0620 18:42:17.654847 (+ 1775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:17.654852 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:17.654922 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:17.655057 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:17.656494 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:17.657235 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:17.658750 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:17.659370 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:17.659484 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:17.663706 (+ 4222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:17.663711 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:17.663714 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:17.663747 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:17.666828 (+ 3081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:17.667569 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:17.667573 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:17.667576 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:17.667690 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:17.668102 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:17.668107 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:17.668111 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:17.668385 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:17.668512 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:17.669237 (+ 725us) server_negotiation.cc:299] Negotiation successful
0620 18:42:17.669486 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 18:42:17.670781 8055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:17.652249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:44779 (local address 127.23.115.3:45845)
0620 18:42:17.652924 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:17.652959 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:17.654551 (+ 1592us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:17.655275 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:17.655283 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:17.655798 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:42:17.656333 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:17.656350 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:17.657376 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:17.657379 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:17.658587 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:17.658599 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:17.659675 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:17.659685 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:17.659980 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:17.660808 (+ 828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:42:17.660832 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:42:17.663495 (+ 2663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:42:17.666980 (+ 3485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:17.666989 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:17.667007 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:17.667456 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:17.667792 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:17.667797 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:17.667800 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:17.667989 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:17.668506 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:42:17.668514 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:42:17.668869 (+ 355us) client_negotiation.cc:769] Sending connection context
0620 18:42:17.669121 (+ 252us) client_negotiation.cc:240] Negotiation successful
0620 18:42:17.669662 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":269,"threads_started":1}
I20260620 18:42:17.672189 8053 heartbeater.cc:347] Connected to a master server at 127.23.115.62:44779
I20260620 18:42:17.672469 8053 heartbeater.cc:464] Registering TS with master...
I20260620 18:42:17.673168 8053 heartbeater.cc:511] Master 127.23.115.62:44779 requested a full tablet report, sending...
I20260620 18:42:17.674186 8056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:17.659801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35228 (local address 127.23.115.62:44779)
0620 18:42:17.660755 (+ 954us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:17.660760 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:17.660874 (+ 114us) server_negotiation.cc:407] Connection header received
0620 18:42:17.661011 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:17.661016 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:17.661097 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:17.661205 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:17.662875 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:17.663724 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:17.665446 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:17.666158 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:17.666295 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:17.669130 (+ 2835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:17.669135 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:17.669138 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:17.669168 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:17.671741 (+ 2573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:17.672365 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:17.672371 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:17.672374 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:17.672486 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:17.672844 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:17.672849 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:17.672853 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:17.673102 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:17.673323 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:17.673624 (+ 301us) server_negotiation.cc:299] Negotiation successful
0620 18:42:17.673821 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":841,"thread_start_us":704,"threads_started":1}
I20260620 18:42:17.674544 7605 ts_manager.cc:194] Registered new tserver with Master: 8205483c83dc4727a92ef655e516de62 (127.23.115.3:38983)
I20260620 18:42:17.675593 7605 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.115.3@KRBTEST.COM'} at 127.23.115.3:45845
I20260620 18:42:17.676604 24012 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:42:17.682801 8056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:17.677444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35230 (local address 127.23.115.62:44779)
0620 18:42:17.677608 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:17.677613 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:17.678085 (+ 472us) server_negotiation.cc:407] Connection header received
0620 18:42:17.678230 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:17.678233 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:17.678370 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:17.678476 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:42:17.678872 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:17.679591 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:17.680972 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:17.681470 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:17.681716 (+ 246us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:17.682131 (+ 415us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:42:17.682321 (+ 190us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0620 18:42:17.682572 (+ 251us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35230: token expired
Metrics: {"server-negotiator.queue_time_us":56}
W20260620 18:42:17.682964 8056 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35230: token expired
W20260620 18:42:17.683519 7527 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.23.115.62:44779)
I20260620 18:42:17.695797 8056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:17.683957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35240 (local address 127.23.115.62:44779)
0620 18:42:17.684090 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:17.684094 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:17.684108 (+ 14us) server_negotiation.cc:407] Connection header received
0620 18:42:17.684181 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:17.684184 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:17.684239 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:17.684322 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:17.685551 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:17.686265 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:17.687890 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:17.688365 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:17.688456 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:17.691070 (+ 2614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:17.691075 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:17.691078 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:17.691108 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:17.693388 (+ 2280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:17.694045 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:17.694049 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:17.694051 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:17.694173 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:17.694541 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:17.694544 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:17.694546 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:17.694838 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:17.695059 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:17.695356 (+ 297us) server_negotiation.cc:299] Negotiation successful
0620 18:42:17.695515 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 18:42:17.699733 7604 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35240:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:42:17.707547 7604 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:42:17.731191 8064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:17.719469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.3:38983 (local address 127.0.0.1:42156)
0620 18:42:17.720765 (+ 1296us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:17.720804 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:17.720978 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:17.722025 (+ 1047us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:17.722032 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:17.722061 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:42:17.722499 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:17.722512 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:17.724699 (+ 2187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:17.724704 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:17.728229 (+ 3525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:17.728242 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:17.730058 (+ 1816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:17.730062 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:17.730254 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:17.730395 (+ 141us) client_negotiation.cc:769] Sending connection context
0620 18:42:17.730525 (+ 130us) client_negotiation.cc:240] Negotiation successful
0620 18:42:17.730732 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1075,"mutex_wait_us":1,"thread_start_us":927,"threads_started":1}
I20260620 18:42:17.731168 8063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:17.719622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42156 (local address 127.23.115.3:38983)
0620 18:42:17.720442 (+ 820us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:17.720451 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:17.720914 (+ 463us) server_negotiation.cc:407] Connection header received
0620 18:42:17.721280 (+ 366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:17.721289 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:17.721523 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:17.721688 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:42:17.722753 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:17.724488 (+ 1735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:17.728398 (+ 3910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:17.729577 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:17.729679 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:17.729872 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:17.730593 (+ 721us) server_negotiation.cc:299] Negotiation successful
0620 18:42:17.730783 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":632,"thread_start_us":460,"threads_started":1}
I20260620 18:42:17.732069 8066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:17.720026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60688 (local address 127.23.115.1:46605)
0620 18:42:17.720826 (+ 800us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:17.720832 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:17.721350 (+ 518us) server_negotiation.cc:407] Connection header received
0620 18:42:17.721669 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:17.721678 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:17.721905 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:17.722059 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:42:17.723126 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:17.725999 (+ 2873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:17.728727 (+ 2728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:17.730369 (+ 1642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:17.730493 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:17.730684 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:17.731529 (+ 845us) server_negotiation.cc:299] Negotiation successful
0620 18:42:17.731681 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":666,"thread_start_us":566,"threads_started":1}
I20260620 18:42:17.732867 8065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:17.719546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.1:46605 (local address 127.0.0.1:60688)
0620 18:42:17.721238 (+ 1692us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:17.721260 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:17.721462 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:17.722468 (+ 1006us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:17.722477 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:17.722505 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:42:17.722923 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:17.722954 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:17.726184 (+ 3230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:17.726189 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:17.728538 (+ 2349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:17.728555 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:17.730953 (+ 2398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:17.730959 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:17.731161 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:17.731201 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 18:42:17.731288 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 18:42:17.732390 (+ 1102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1570,"mutex_wait_us":23,"thread_start_us":1731,"threads_started":1}
I20260620 18:42:17.736930 7725 tablet_service.cc:1467] Processing CreateTablet for tablet 6ecb30107c224be8849ade73568efb24 (DEFAULT_TABLE table=test-table [id=a3e5c95ebc0a4fbc926c40f09d607d94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:42:17.736930 7988 tablet_service.cc:1467] Processing CreateTablet for tablet 6ecb30107c224be8849ade73568efb24 (DEFAULT_TABLE table=test-table [id=a3e5c95ebc0a4fbc926c40f09d607d94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:42:17.738382 7988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ecb30107c224be8849ade73568efb24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:42:17.738384 7725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ecb30107c224be8849ade73568efb24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:42:17.738778 8064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:17.721172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.2:35821 (local address 127.0.0.1:43906)
0620 18:42:17.731491 (+ 10319us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:17.731523 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:17.731633 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:17.733387 (+ 1754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:17.733391 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:17.733417 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:42:17.733575 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:17.733586 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:17.735072 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:17.735078 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:17.736276 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:17.736289 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:17.737962 (+ 1673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:17.737969 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:17.738167 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:17.738206 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 18:42:17.738303 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 18:42:17.738454 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10231,"thread_start_us":12053,"threads_started":1}
I20260620 18:42:17.739636 8068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:17.721514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43906 (local address 127.23.115.2:35821)
0620 18:42:17.727562 (+ 6048us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:17.727567 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:17.731755 (+ 4188us) server_negotiation.cc:407] Connection header received
0620 18:42:17.731859 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:17.731867 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:17.732125 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:17.732489 (+ 364us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:42:17.733735 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:17.734902 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:17.736447 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:17.737498 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:17.737656 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:17.737854 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:17.739249 (+ 1395us) server_negotiation.cc:299] Negotiation successful
0620 18:42:17.739391 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5899,"thread_start_us":5775,"threads_started":1}
I20260620 18:42:17.742502 7857 tablet_service.cc:1467] Processing CreateTablet for tablet 6ecb30107c224be8849ade73568efb24 (DEFAULT_TABLE table=test-table [id=a3e5c95ebc0a4fbc926c40f09d607d94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:42:17.743813 7857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ecb30107c224be8849ade73568efb24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:42:17.760406 8069 tablet_bootstrap.cc:492] T 6ecb30107c224be8849ade73568efb24 P 86629c1da49b4263ae52078ef6a60192: Bootstrap starting.
I20260620 18:42:17.766638 8070 tablet_bootstrap.cc:492] T 6ecb30107c224be8849ade73568efb24 P 8205483c83dc4727a92ef655e516de62: Bootstrap starting.
I20260620 18:42:17.768579 8069 tablet_bootstrap.cc:654] T 6ecb30107c224be8849ade73568efb24 P 86629c1da49b4263ae52078ef6a60192: Neither blocks nor log segments found. Creating new log.
I20260620 18:42:17.768908 8071 tablet_bootstrap.cc:492] T 6ecb30107c224be8849ade73568efb24 P 5de3fa128ee24886be30c46b379343ba: Bootstrap starting.
I20260620 18:42:17.769891 8069 log.cc:824] T 6ecb30107c224be8849ade73568efb24 P 86629c1da49b4263ae52078ef6a60192: Log is configured to *not* fsync() on all Append() calls
I20260620 18:42:17.770704 8070 tablet_bootstrap.cc:654] T 6ecb30107c224be8849ade73568efb24 P 8205483c83dc4727a92ef655e516de62: Neither blocks nor log segments found. Creating new log.
I20260620 18:42:17.771809 8070 log.cc:824] T 6ecb30107c224be8849ade73568efb24 P 8205483c83dc4727a92ef655e516de62: Log is configured to *not* fsync() on all Append() calls
I20260620 18:42:17.772255 8071 tablet_bootstrap.cc:654] T 6ecb30107c224be8849ade73568efb24 P 5de3fa128ee24886be30c46b379343ba: Neither blocks nor log segments found. Creating new log.
I20260620 18:42:17.773087 8069 tablet_bootstrap.cc:492] T 6ecb30107c224be8849ade73568efb24 P 86629c1da49b4263ae52078ef6a60192: No bootstrap required, opened a new log
I20260620 18:42:17.773377 8069 ts_tablet_manager.cc:1397] T 6ecb30107c224be8849ade73568efb24 P 86629c1da49b4263ae52078ef6a60192: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.007s
I20260620 18:42:17.773440 8071 log.cc:824] T 6ecb30107c224be8849ade73568efb24 P 5de3fa128ee24886be30c46b379343ba: Log is configured to *not* fsync() on all Append() calls
I20260620 18:42:17.774873 8070 tablet_bootstrap.cc:492] T 6ecb30107c224be8849ade73568efb24 P 8205483c83dc4727a92ef655e516de62: No bootstrap required, opened a new log
I20260620 18:42:17.775159 8070 ts_tablet_manager.cc:1397] T 6ecb30107c224be8849ade73568efb24 P 8205483c83dc4727a92ef655e516de62: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260620 18:42:17.776302 8071 tablet_bootstrap.cc:492] T 6ecb30107c224be8849ade73568efb24 P 5de3fa128ee24886be30c46b379343ba: No bootstrap required, opened a new log
I20260620 18:42:17.776554 8071 ts_tablet_manager.cc:1397] T 6ecb30107c224be8849ade73568efb24 P 5de3fa128ee24886be30c46b379343ba: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.001s
I20260620 18:42:17.783715 8071 raft_consensus.cc:348] T 6ecb30107c224be8849ade73568efb24 P 5de3fa128ee24886be30c46b379343ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8205483c83dc4727a92ef655e516de62" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 38983 } } peers { permanent_uuid: "86629c1da49b4263ae52078ef6a60192" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 46605 } } peers { permanent_uuid: "5de3fa128ee24886be30c46b379343ba" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 35821 } }
I20260620 18:42:17.784090 8071 raft_consensus.cc:374] T 6ecb30107c224be8849ade73568efb24 P 5de3fa128ee24886be30c46b379343ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:42:17.784185 8071 raft_consensus.cc:729] T 6ecb30107c224be8849ade73568efb24 P 5de3fa128ee24886be30c46b379343ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5de3fa128ee24886be30c46b379343ba, State: Initialized, Role: FOLLOWER
I20260620 18:42:17.784277 8069 raft_consensus.cc:348] T 6ecb30107c224be8849ade73568efb24 P 86629c1da49b4263ae52078ef6a60192 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8205483c83dc4727a92ef655e516de62" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 38983 } } peers { permanent_uuid: "86629c1da49b4263ae52078ef6a60192" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 46605 } } peers { permanent_uuid: "5de3fa128ee24886be30c46b379343ba" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 35821 } }
I20260620 18:42:17.784669 8069 raft_consensus.cc:374] T 6ecb30107c224be8849ade73568efb24 P 86629c1da49b4263ae52078ef6a60192 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:42:17.784780 8069 raft_consensus.cc:729] T 6ecb30107c224be8849ade73568efb24 P 86629c1da49b4263ae52078ef6a60192 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86629c1da49b4263ae52078ef6a60192, State: Initialized, Role: FOLLOWER
I20260620 18:42:17.784832 8071 consensus_queue.cc:260] T 6ecb30107c224be8849ade73568efb24 P 5de3fa128ee24886be30c46b379343ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8205483c83dc4727a92ef655e516de62" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 38983 } } peers { permanent_uuid: "86629c1da49b4263ae52078ef6a60192" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 46605 } } peers { permanent_uuid: "5de3fa128ee24886be30c46b379343ba" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 35821 } }
I20260620 18:42:17.785162 8070 raft_consensus.cc:348] T 6ecb30107c224be8849ade73568efb24 P 8205483c83dc4727a92ef655e516de62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8205483c83dc4727a92ef655e516de62" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 38983 } } peers { permanent_uuid: "86629c1da49b4263ae52078ef6a60192" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 46605 } } peers { permanent_uuid: "5de3fa128ee24886be30c46b379343ba" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 35821 } }
I20260620 18:42:17.785372 8069 consensus_queue.cc:260] T 6ecb30107c224be8849ade73568efb24 P 86629c1da49b4263ae52078ef6a60192 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8205483c83dc4727a92ef655e516de62" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 38983 } } peers { permanent_uuid: "86629c1da49b4263ae52078ef6a60192" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 46605 } } peers { permanent_uuid: "5de3fa128ee24886be30c46b379343ba" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 35821 } }
I20260620 18:42:17.785575 8070 raft_consensus.cc:374] T 6ecb30107c224be8849ade73568efb24 P 8205483c83dc4727a92ef655e516de62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:42:17.785660 8070 raft_consensus.cc:729] T 6ecb30107c224be8849ade73568efb24 P 8205483c83dc4727a92ef655e516de62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8205483c83dc4727a92ef655e516de62, State: Initialized, Role: FOLLOWER
I20260620 18:42:17.786235 7922 heartbeater.cc:503] Master 127.23.115.62:44779 was elected leader, sending a full tablet report...
I20260620 18:42:17.786391 8070 consensus_queue.cc:260] T 6ecb30107c224be8849ade73568efb24 P 8205483c83dc4727a92ef655e516de62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8205483c83dc4727a92ef655e516de62" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 38983 } } peers { permanent_uuid: "86629c1da49b4263ae52078ef6a60192" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 46605 } } peers { permanent_uuid: "5de3fa128ee24886be30c46b379343ba" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 35821 } }
I20260620 18:42:17.786747 8071 ts_tablet_manager.cc:1428] T 6ecb30107c224be8849ade73568efb24 P 5de3fa128ee24886be30c46b379343ba: Time spent starting tablet: real 0.010s user 0.007s sys 0.003s
I20260620 18:42:17.787786 7790 heartbeater.cc:503] Master 127.23.115.62:44779 was elected leader, sending a full tablet report...
I20260620 18:42:17.788419 8069 ts_tablet_manager.cc:1428] T 6ecb30107c224be8849ade73568efb24 P 86629c1da49b4263ae52078ef6a60192: Time spent starting tablet: real 0.015s user 0.011s sys 0.005s
I20260620 18:42:17.789299 8053 heartbeater.cc:503] Master 127.23.115.62:44779 was elected leader, sending a full tablet report...
I20260620 18:42:17.789860 8070 ts_tablet_manager.cc:1428] T 6ecb30107c224be8849ade73568efb24 P 8205483c83dc4727a92ef655e516de62: Time spent starting tablet: real 0.015s user 0.009s sys 0.005s
I20260620 18:42:17.878334 8075 raft_consensus.cc:482] T 6ecb30107c224be8849ade73568efb24 P 5de3fa128ee24886be30c46b379343ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:42:17.878620 8075 raft_consensus.cc:504] T 6ecb30107c224be8849ade73568efb24 P 5de3fa128ee24886be30c46b379343ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8205483c83dc4727a92ef655e516de62" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 38983 } } peers { permanent_uuid: "86629c1da49b4263ae52078ef6a60192" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 46605 } } peers { permanent_uuid: "5de3fa128ee24886be30c46b379343ba" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 35821 } }
I20260620 18:42:17.880321 8075 leader_election.cc:288] T 6ecb30107c224be8849ade73568efb24 P 5de3fa128ee24886be30c46b379343ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8205483c83dc4727a92ef655e516de62 (127.23.115.3:38983), 86629c1da49b4263ae52078ef6a60192 (127.23.115.1:46605)
I20260620 18:42:17.887583 8063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:17.880739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.2:45097 (local address 127.23.115.3:38983)
0620 18:42:17.880943 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:17.880949 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:17.882211 (+ 1262us) server_negotiation.cc:407] Connection header received
0620 18:42:17.882418 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:17.882424 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:17.882496 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:17.882560 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:42:17.883165 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:17.883903 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:17.885854 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:17.886629 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:17.886756 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:17.886829 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:17.887218 (+ 389us) server_negotiation.cc:299] Negotiation successful
0620 18:42:17.887337 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 18:42:17.887601 8066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:17.881397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.2:60643 (local address 127.23.115.1:46605)
0620 18:42:17.881594 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:17.881600 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:17.882314 (+ 714us) server_negotiation.cc:407] Connection header received
0620 18:42:17.882497 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:17.882502 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:17.882570 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:17.882657 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:42:17.883295 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:17.884020 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:17.885912 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:17.886658 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:17.886825 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:17.886895 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:17.887239 (+ 344us) server_negotiation.cc:299] Negotiation successful
0620 18:42:17.887359 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 18:42:17.887754 8079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:17.880714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.3:38983 (local address 127.23.115.2:45097)
0620 18:42:17.882102 (+ 1388us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:17.882121 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:17.882240 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:17.882622 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:17.882628 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:17.882653 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:42:17.883026 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:17.883036 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:17.884042 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:17.884048 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:17.885707 (+ 1659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:17.885720 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:17.886855 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:17.886865 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:17.887065 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:17.887090 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 18:42:17.887165 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 18:42:17.887295 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1263,"mutex_wait_us":27,"thread_start_us":1093,"threads_started":1}
I20260620 18:42:17.887779 8078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:17.880860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.1:46605 (local address 127.23.115.2:60643)
0620 18:42:17.882214 (+ 1354us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:17.882232 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:17.882361 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:17.882684 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:17.882689 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:17.882714 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:42:17.883143 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:17.883155 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:17.884146 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:17.884149 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:17.885766 (+ 1617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:17.885779 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:17.886880 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:17.886886 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:17.887081 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:17.887103 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 18:42:17.887184 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 18:42:17.887299 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1237,"mutex_wait_us":51,"thread_start_us":911,"threads_started":1}
I20260620 18:42:17.888849 7745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ecb30107c224be8849ade73568efb24" candidate_uuid: "5de3fa128ee24886be30c46b379343ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86629c1da49b4263ae52078ef6a60192" is_pre_election: true
I20260620 18:42:17.888975 8008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ecb30107c224be8849ade73568efb24" candidate_uuid: "5de3fa128ee24886be30c46b379343ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8205483c83dc4727a92ef655e516de62" is_pre_election: true
I20260620 18:42:17.889385 7745 raft_consensus.cc:2454] T 6ecb30107c224be8849ade73568efb24 P 86629c1da49b4263ae52078ef6a60192 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5de3fa128ee24886be30c46b379343ba in term 0.
I20260620 18:42:17.889451 8008 raft_consensus.cc:2454] T 6ecb30107c224be8849ade73568efb24 P 8205483c83dc4727a92ef655e516de62 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5de3fa128ee24886be30c46b379343ba in term 0.
I20260620 18:42:17.890040 7811 leader_election.cc:302] T 6ecb30107c224be8849ade73568efb24 P 5de3fa128ee24886be30c46b379343ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5de3fa128ee24886be30c46b379343ba, 86629c1da49b4263ae52078ef6a60192; no voters:
I20260620 18:42:17.890416 8075 raft_consensus.cc:2781] T 6ecb30107c224be8849ade73568efb24 P 5de3fa128ee24886be30c46b379343ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:42:17.890563 8075 raft_consensus.cc:482] T 6ecb30107c224be8849ade73568efb24 P 5de3fa128ee24886be30c46b379343ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:42:17.890662 8075 raft_consensus.cc:3037] T 6ecb30107c224be8849ade73568efb24 P 5de3fa128ee24886be30c46b379343ba [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:42:17.892789 8075 raft_consensus.cc:504] T 6ecb30107c224be8849ade73568efb24 P 5de3fa128ee24886be30c46b379343ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8205483c83dc4727a92ef655e516de62" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 38983 } } peers { permanent_uuid: "86629c1da49b4263ae52078ef6a60192" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 46605 } } peers { permanent_uuid: "5de3fa128ee24886be30c46b379343ba" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 35821 } }
I20260620 18:42:17.893312 8075 leader_election.cc:288] T 6ecb30107c224be8849ade73568efb24 P 5de3fa128ee24886be30c46b379343ba [CANDIDATE]: Term 1 election: Requested vote from peers 8205483c83dc4727a92ef655e516de62 (127.23.115.3:38983), 86629c1da49b4263ae52078ef6a60192 (127.23.115.1:46605)
I20260620 18:42:17.893766 7745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ecb30107c224be8849ade73568efb24" candidate_uuid: "5de3fa128ee24886be30c46b379343ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86629c1da49b4263ae52078ef6a60192"
I20260620 18:42:17.893766 8008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ecb30107c224be8849ade73568efb24" candidate_uuid: "5de3fa128ee24886be30c46b379343ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8205483c83dc4727a92ef655e516de62"
I20260620 18:42:17.893967 8008 raft_consensus.cc:3037] T 6ecb30107c224be8849ade73568efb24 P 8205483c83dc4727a92ef655e516de62 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:42:17.893967 7745 raft_consensus.cc:3037] T 6ecb30107c224be8849ade73568efb24 P 86629c1da49b4263ae52078ef6a60192 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:42:17.897106 8008 raft_consensus.cc:2454] T 6ecb30107c224be8849ade73568efb24 P 8205483c83dc4727a92ef655e516de62 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5de3fa128ee24886be30c46b379343ba in term 1.
I20260620 18:42:17.897106 7745 raft_consensus.cc:2454] T 6ecb30107c224be8849ade73568efb24 P 86629c1da49b4263ae52078ef6a60192 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5de3fa128ee24886be30c46b379343ba in term 1.
I20260620 18:42:17.897519 7813 leader_election.cc:302] T 6ecb30107c224be8849ade73568efb24 P 5de3fa128ee24886be30c46b379343ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5de3fa128ee24886be30c46b379343ba, 8205483c83dc4727a92ef655e516de62; no voters:
I20260620 18:42:17.897799 8075 raft_consensus.cc:2781] T 6ecb30107c224be8849ade73568efb24 P 5de3fa128ee24886be30c46b379343ba [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:42:17.898191 8075 raft_consensus.cc:686] T 6ecb30107c224be8849ade73568efb24 P 5de3fa128ee24886be30c46b379343ba [term 1 LEADER]: Becoming Leader. State: Replica: 5de3fa128ee24886be30c46b379343ba, State: Running, Role: LEADER
I20260620 18:42:17.898654 8075 consensus_queue.cc:237] T 6ecb30107c224be8849ade73568efb24 P 5de3fa128ee24886be30c46b379343ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8205483c83dc4727a92ef655e516de62" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 38983 } } peers { permanent_uuid: "86629c1da49b4263ae52078ef6a60192" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 46605 } } peers { permanent_uuid: "5de3fa128ee24886be30c46b379343ba" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 35821 } }
W20260620 18:42:17.900852 8054 tablet.cc:2369] T 6ecb30107c224be8849ade73568efb24 P 8205483c83dc4727a92ef655e516de62: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:42:17.903206 7603 catalog_manager.cc:5495] T 6ecb30107c224be8849ade73568efb24 P 5de3fa128ee24886be30c46b379343ba reported cstate change: term changed from 0 to 1, leader changed from <none> to 5de3fa128ee24886be30c46b379343ba (127.23.115.2). New cstate: current_term: 1 leader_uuid: "5de3fa128ee24886be30c46b379343ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8205483c83dc4727a92ef655e516de62" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 38983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "86629c1da49b4263ae52078ef6a60192" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 46605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5de3fa128ee24886be30c46b379343ba" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 35821 } health_report { overall_health: HEALTHY } } }
I20260620 18:42:17.930368 8068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:17.925034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43920 (local address 127.23.115.2:35821)
0620 18:42:17.925197 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:17.925216 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:17.925246 (+ 30us) server_negotiation.cc:407] Connection header received
0620 18:42:17.925382 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:17.925385 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:17.925468 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:17.925587 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:42:17.925978 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:17.926700 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:17.928199 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:17.928622 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:17.928722 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:17.929230 (+ 508us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:42:17.929760 (+ 530us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:42:17.929890 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:17.930001 (+ 111us) server_negotiation.cc:299] Negotiation successful
0620 18:42:17.930072 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 18:42:17.938510 7745 raft_consensus.cc:1261] T 6ecb30107c224be8849ade73568efb24 P 86629c1da49b4263ae52078ef6a60192 [term 1 FOLLOWER]: Refusing update from remote peer 5de3fa128ee24886be30c46b379343ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:42:17.938621 8008 raft_consensus.cc:1261] T 6ecb30107c224be8849ade73568efb24 P 8205483c83dc4727a92ef655e516de62 [term 1 FOLLOWER]: Refusing update from remote peer 5de3fa128ee24886be30c46b379343ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:42:17.939605 8080 consensus_queue.cc:1035] T 6ecb30107c224be8849ade73568efb24 P 5de3fa128ee24886be30c46b379343ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "86629c1da49b4263ae52078ef6a60192" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 46605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:42:17.939975 8075 consensus_queue.cc:1035] T 6ecb30107c224be8849ade73568efb24 P 5de3fa128ee24886be30c46b379343ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "8205483c83dc4727a92ef655e516de62" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 38983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:42:17.953150 8085 mvcc.cc:205] Tried to move back new op lower bound from 7298993921785733120 to 7298993921636544512. Current Snapshot: MvccSnapshot[applied={T|T < 7298993921785733120}]
I20260620 18:42:17.957922 8084 mvcc.cc:205] Tried to move back new op lower bound from 7298993921785733120 to 7298993921636544512. Current Snapshot: MvccSnapshot[applied={T|T < 7298993921785733120}]
I20260620 18:42:17.963958 7604 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35240:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:42:17.964219 7604 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35240:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:42:17.967603 7604 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2ea2131d7e994f00a8a2e0f24724e171: Sending DeleteTablet for 3 replicas of tablet 6ecb30107c224be8849ade73568efb24
I20260620 18:42:17.968760 7725 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ecb30107c224be8849ade73568efb24 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:42:17 UTC) from {username='kudu', principal='kudu/127.23.115.62@KRBTEST.COM'} at 127.0.0.1:60688
I20260620 18:42:17.968773 7988 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ecb30107c224be8849ade73568efb24 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:42:17 UTC) from {username='kudu', principal='kudu/127.23.115.62@KRBTEST.COM'} at 127.0.0.1:42156
I20260620 18:42:17.968806 7857 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ecb30107c224be8849ade73568efb24 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:42:17 UTC) from {username='kudu', principal='kudu/127.23.115.62@KRBTEST.COM'} at 127.0.0.1:43906
I20260620 18:42:17.969949 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 7642
I20260620 18:42:17.980731 8094 tablet_replica.cc:330] T 6ecb30107c224be8849ade73568efb24 P 86629c1da49b4263ae52078ef6a60192: stopping tablet replica
I20260620 18:42:17.981328 8094 raft_consensus.cc:2229] T 6ecb30107c224be8849ade73568efb24 P 86629c1da49b4263ae52078ef6a60192 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:42:17.981932 8094 raft_consensus.cc:2258] T 6ecb30107c224be8849ade73568efb24 P 86629c1da49b4263ae52078ef6a60192 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:42:17.984225 8094 ts_tablet_manager.cc:1905] T 6ecb30107c224be8849ade73568efb24 P 86629c1da49b4263ae52078ef6a60192: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:42:17.984362 8092 tablet_replica.cc:330] T 6ecb30107c224be8849ade73568efb24 P 8205483c83dc4727a92ef655e516de62: stopping tablet replica
I20260620 18:42:17.984879 8092 raft_consensus.cc:2229] T 6ecb30107c224be8849ade73568efb24 P 8205483c83dc4727a92ef655e516de62 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:42:17.985399 8092 raft_consensus.cc:2258] T 6ecb30107c224be8849ade73568efb24 P 8205483c83dc4727a92ef655e516de62 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:42:17.986222 8093 tablet_replica.cc:330] T 6ecb30107c224be8849ade73568efb24 P 5de3fa128ee24886be30c46b379343ba: stopping tablet replica
I20260620 18:42:17.986637 8093 raft_consensus.cc:2229] T 6ecb30107c224be8849ade73568efb24 P 5de3fa128ee24886be30c46b379343ba [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:42:17.987358 8093 raft_consensus.cc:2258] T 6ecb30107c224be8849ade73568efb24 P 5de3fa128ee24886be30c46b379343ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:42:17.990177 8093 ts_tablet_manager.cc:1905] T 6ecb30107c224be8849ade73568efb24 P 5de3fa128ee24886be30c46b379343ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:42:17.992482 8094 ts_tablet_manager.cc:1918] T 6ecb30107c224be8849ade73568efb24 P 86629c1da49b4263ae52078ef6a60192: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:42:17.992707 8094 log.cc:1196] T 6ecb30107c224be8849ade73568efb24 P 86629c1da49b4263ae52078ef6a60192: Deleting WAL directory at /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-0/wal/wals/6ecb30107c224be8849ade73568efb24
I20260620 18:42:17.993191 8066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:17.971462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60694 (local address 127.23.115.1:46605)
0620 18:42:17.971637 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:17.971644 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:17.971662 (+ 18us) server_negotiation.cc:407] Connection header received
0620 18:42:17.971733 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:17.971739 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:17.971849 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:17.971944 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:17.973934 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:17.975235 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:17.977338 (+ 2103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:17.978048 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:17.978162 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:17.984303 (+ 6141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:17.984320 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:17.984337 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:17.984380 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:17.988420 (+ 4040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:17.989798 (+ 1378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:17.989807 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:17.989811 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:17.989974 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:17.990475 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:17.990480 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:17.990483 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:17.991058 (+ 575us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:17.991345 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:17.992554 (+ 1209us) server_negotiation.cc:299] Negotiation successful
0620 18:42:17.992816 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 18:42:17.993578 8094 ts_tablet_manager.cc:1939] T 6ecb30107c224be8849ade73568efb24 P 86629c1da49b4263ae52078ef6a60192: Deleting consensus metadata
I20260620 18:42:17.993824 7785 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:42:17.997824 8093 ts_tablet_manager.cc:1918] T 6ecb30107c224be8849ade73568efb24 P 5de3fa128ee24886be30c46b379343ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:42:17.998014 8093 log.cc:1196] T 6ecb30107c224be8849ade73568efb24 P 5de3fa128ee24886be30c46b379343ba: Deleting WAL directory at /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-1/wal/wals/6ecb30107c224be8849ade73568efb24
I20260620 18:42:17.998561 8093 ts_tablet_manager.cc:1939] T 6ecb30107c224be8849ade73568efb24 P 5de3fa128ee24886be30c46b379343ba: Deleting consensus metadata
I20260620 18:42:18.000715 8092 ts_tablet_manager.cc:1905] T 6ecb30107c224be8849ade73568efb24 P 8205483c83dc4727a92ef655e516de62: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:42:18.007180 7592 catalog_manager.cc:4841] TS 5de3fa128ee24886be30c46b379343ba (127.23.115.2:35821): tablet 6ecb30107c224be8849ade73568efb24 (table test-table [id=a3e5c95ebc0a4fbc926c40f09d607d94]) successfully deleted
I20260620 18:42:18.009065 8092 ts_tablet_manager.cc:1918] T 6ecb30107c224be8849ade73568efb24 P 8205483c83dc4727a92ef655e516de62: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:42:18.009251 8092 log.cc:1196] T 6ecb30107c224be8849ade73568efb24 P 8205483c83dc4727a92ef655e516de62: Deleting WAL directory at /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-2/wal/wals/6ecb30107c224be8849ade73568efb24
I20260620 18:42:18.009780 8092 ts_tablet_manager.cc:1939] T 6ecb30107c224be8849ade73568efb24 P 8205483c83dc4727a92ef655e516de62: Deleting consensus metadata
I20260620 18:42:18.011329 7592 catalog_manager.cc:4841] TS 8205483c83dc4727a92ef655e516de62 (127.23.115.3:38983): tablet 6ecb30107c224be8849ade73568efb24 (table test-table [id=a3e5c95ebc0a4fbc926c40f09d607d94]) successfully deleted
I20260620 18:42:18.100193 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 7642
I20260620 18:42:18.100481 7590 catalog_manager.cc:4841] TS 86629c1da49b4263ae52078ef6a60192 (127.23.115.1:46605): tablet 6ecb30107c224be8849ade73568efb24 (table test-table [id=a3e5c95ebc0a4fbc926c40f09d607d94]) successfully deleted
I20260620 18:42:18.116480 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 7794
I20260620 18:42:18.137598 8068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:18.117253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43928 (local address 127.23.115.2:35821)
0620 18:42:18.117425 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:18.117429 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:18.117466 (+ 37us) server_negotiation.cc:407] Connection header received
0620 18:42:18.117636 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:18.117676 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:18.117752 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:18.117836 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:18.119555 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:18.120360 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:18.122180 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:18.122610 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:18.122700 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:18.130599 (+ 7899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:18.130619 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:18.130635 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:18.130712 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:18.133903 (+ 3191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:18.134873 (+ 970us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:18.134887 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:18.134890 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:18.135520 (+ 630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:18.135939 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:18.135943 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:18.135945 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:18.136365 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:18.136635 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:18.137008 (+ 373us) server_negotiation.cc:299] Negotiation successful
0620 18:42:18.137195 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 18:42:18.138599 7917 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:42:18.252279 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 7794
I20260620 18:42:18.268229 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 7925
I20260620 18:42:18.284521 8063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:18.268983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42160 (local address 127.23.115.3:38983)
0620 18:42:18.269167 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:18.269171 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:18.269191 (+ 20us) server_negotiation.cc:407] Connection header received
0620 18:42:18.269379 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:18.269385 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:18.269473 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:18.269611 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:18.271298 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:18.272037 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:18.273800 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:18.274194 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:18.274309 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:18.277639 (+ 3330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:18.277654 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:18.277671 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:18.277705 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:18.280443 (+ 2738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:18.282041 (+ 1598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:18.282050 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:18.282052 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:18.282233 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:18.282677 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:18.282681 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:18.282683 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:18.283138 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:18.283540 (+ 402us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:18.283739 (+ 199us) server_negotiation.cc:299] Negotiation successful
0620 18:42:18.284224 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 18:42:18.285104 8048 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:42:18.374629 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 7925
I20260620 18:42:18.388948 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 7572
I20260620 18:42:18.390173 7635 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:42:18.501211 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 7572
I20260620 18:42:38.519977 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.115.62:44779
--webserver_interface=127.23.115.62
--webserver_port=36309
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab
--principal=kudu/127.23.115.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:41715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.115.62:44779
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:42:38.920504 8099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:42:38.920902 8099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:42:38.921047 8099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:42:38.929349 8099 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 18:42:38.929494 8099 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 18:42:38.929548 8099 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 18:42:38.929630 8099 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:42:38.929701 8099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:42:38.929802 8099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:42:38.929874 8099 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:42:38.929942 8099 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:42:38.941618 8099 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:41715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.23.115.62:44779
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab
--principal=kudu/127.23.115.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.115.62:44779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.115.62
--webserver_port=36309
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:42:38.943463 8099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:42:38.945216 8099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:42:38.959867 8104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:38.959967 8105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:38.959913 8107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:42:38.960891 8099 server_base.cc:1034] running on GCE node
I20260620 18:42:38.961958 8099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:42:38.963961 8099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:42:38.965270 8099 hybrid_clock.cc:648] HybridClock initialized: now 1781980958965174 us; error 53 us; skew 500 ppm
I20260620 18:42:38.965798 8099 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:42:38 dist-test-slave-d645 krb5kdc[7020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980958, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:42:38.969781 8099 init.cc:377] Logged in from keytab as kudu/127.23.115.62@KRBTEST.COM (short username kudu)
I20260620 18:42:38.972113 8099 webserver.cc:460] Webserver started at http://127.23.115.62:36309/ using document root <none> and password file <none>
I20260620 18:42:38.973105 8099 fs_manager.cc:359] Metadata directory not provided
I20260620 18:42:38.973251 8099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:42:38.980609 8099 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 18:42:38.984668 8114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:38.986531 8099 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260620 18:42:38.986828 8099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/master-0/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/master-0/wal
uuid: "2ea2131d7e994f00a8a2e0f24724e171"
format_stamp: "Formatted at 2026-06-20 18:41:52 on dist-test-slave-d645"
server_key: "d89fe0e68734db40a03bc14e40aaf1c6"
server_key_iv: "8b96b28491fa7104a042e84a15200d10"
server_key_version: "encryptionkey@0"
I20260620 18:42:38.988094 8099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:42:39.004964 8099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:42:39.011600 8099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:42:39.012106 8099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:42:39.036000 8099 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.62:44779
I20260620 18:42:39.036024 8165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.62:44779 every 8 connection(s)
I20260620 18:42:39.037962 8099 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
I20260620 18:42:39.038662 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 8099
I20260620 18:42:39.039686 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.115.1:46605
--local_ip_for_outbound_sockets=127.23.115.1
--tserver_master_addrs=127.23.115.62:44779
--webserver_port=35971
--webserver_interface=127.23.115.1
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.1.keytab
--principal=kudu/127.23.115.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:41715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:42:39.047410 8166 sys_catalog.cc:263] Verifying existing consensus state
I20260620 18:42:39.052989 8166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ea2131d7e994f00a8a2e0f24724e171: Bootstrap starting.
I20260620 18:42:39.089504 8166 log.cc:824] T 00000000000000000000000000000000 P 2ea2131d7e994f00a8a2e0f24724e171: Log is configured to *not* fsync() on all Append() calls
I20260620 18:42:39.114823 8166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ea2131d7e994f00a8a2e0f24724e171: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 18:42:39.115700 8166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ea2131d7e994f00a8a2e0f24724e171: Bootstrap complete.
I20260620 18:42:39.131707 8166 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ea2131d7e994f00a8a2e0f24724e171 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ea2131d7e994f00a8a2e0f24724e171" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 44779 } }
I20260620 18:42:39.133230 8166 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ea2131d7e994f00a8a2e0f24724e171 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ea2131d7e994f00a8a2e0f24724e171, State: Initialized, Role: FOLLOWER
I20260620 18:42:39.134192 8166 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ea2131d7e994f00a8a2e0f24724e171 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ea2131d7e994f00a8a2e0f24724e171" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 44779 } }
I20260620 18:42:39.134554 8166 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2ea2131d7e994f00a8a2e0f24724e171 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:42:39.134697 8166 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ea2131d7e994f00a8a2e0f24724e171 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:42:39.134891 8166 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ea2131d7e994f00a8a2e0f24724e171 [term 2 FOLLOWER]: Advancing to term 3
I20260620 18:42:39.141813 8166 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ea2131d7e994f00a8a2e0f24724e171 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ea2131d7e994f00a8a2e0f24724e171" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 44779 } }
I20260620 18:42:39.142655 8166 leader_election.cc:302] T 00000000000000000000000000000000 P 2ea2131d7e994f00a8a2e0f24724e171 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ea2131d7e994f00a8a2e0f24724e171; no voters:
I20260620 18:42:39.143623 8166 leader_election.cc:288] T 00000000000000000000000000000000 P 2ea2131d7e994f00a8a2e0f24724e171 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 18:42:39.143930 8170 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2ea2131d7e994f00a8a2e0f24724e171 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 18:42:39.144979 8170 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2ea2131d7e994f00a8a2e0f24724e171 [term 3 LEADER]: Becoming Leader. State: Replica: 2ea2131d7e994f00a8a2e0f24724e171, State: Running, Role: LEADER
I20260620 18:42:39.145723 8170 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ea2131d7e994f00a8a2e0f24724e171 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ea2131d7e994f00a8a2e0f24724e171" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 44779 } }
I20260620 18:42:39.146201 8166 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ea2131d7e994f00a8a2e0f24724e171 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:42:39.148742 8172 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ea2131d7e994f00a8a2e0f24724e171 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ea2131d7e994f00a8a2e0f24724e171. Latest consensus state: current_term: 3 leader_uuid: "2ea2131d7e994f00a8a2e0f24724e171" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ea2131d7e994f00a8a2e0f24724e171" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 44779 } } }
I20260620 18:42:39.149171 8172 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ea2131d7e994f00a8a2e0f24724e171 [sys.catalog]: This master's current role is: LEADER
I20260620 18:42:39.148983 8171 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ea2131d7e994f00a8a2e0f24724e171 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "2ea2131d7e994f00a8a2e0f24724e171" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ea2131d7e994f00a8a2e0f24724e171" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 44779 } } }
I20260620 18:42:39.149663 8171 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ea2131d7e994f00a8a2e0f24724e171 [sys.catalog]: This master's current role is: LEADER
I20260620 18:42:39.151469 8176 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:42:39.161002 8176 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:42:39.162732 8176 catalog_manager.cc:1259] Loaded cluster ID: a10c99f42da74d3bbf169911bfe9f448
I20260620 18:42:39.162922 8176 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:42:39.167719 8176 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:42:39.169571 8176 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2ea2131d7e994f00a8a2e0f24724e171: Loaded TSK: 0
I20260620 18:42:39.169795 8176 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2ea2131d7e994f00a8a2e0f24724e171: Loaded TSK: 1
F20260620 18:42:39.171057 8176 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f89000d89ad google::LogMessage::Fail() at ??:0
@ 0x7f89000db88a google::LogMessage::SendToLog() at ??:0
@ 0x7f89000d84de google::LogMessage::Flush() at ??:0
@ 0x7f89000d9a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f8903e3c7d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f8903e2b73e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f8915b071c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f8915b06783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f8915b922a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f8915b920c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f8915bdc5e9 std::function<>::operator()() at ??:0
@ 0x7f8915b0acdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f8915b08c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f8915b90610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f8915b90439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f8902d24345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f8902d2791c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f8902d27799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f8902d02eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f8905fdc6db start_thread at ??:0
@ 0x7f88fef7261f clone at ??:0
W20260620 18:42:39.470834 8168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:42:39.471211 8168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:42:39.471376 8168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:42:39.479152 8168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:42:39.479279 8168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:42:39.479434 8168 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.115.1
I20260620 18:42:39.489524 8168 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:41715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.1.keytab
--principal=kudu/127.23.115.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.115.1:46605
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.115.1
--webserver_port=35971
--tserver_master_addrs=127.23.115.62:44779
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.115.1
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:42:39.490869 8168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:42:39.492180 8168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:42:39.505021 8213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:39.504940 8214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:39.504966 8216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:42:39.505858 8168 server_base.cc:1034] running on GCE node
I20260620 18:42:39.506405 8168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:42:39.507361 8168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:42:39.508567 8168 hybrid_clock.cc:648] HybridClock initialized: now 1781980959508512 us; error 42 us; skew 500 ppm
I20260620 18:42:39.508895 8168 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:42:39 dist-test-slave-d645 krb5kdc[7020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980959, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:42:39.512355 8168 init.cc:377] Logged in from keytab as kudu/127.23.115.1@KRBTEST.COM (short username kudu)
I20260620 18:42:39.513801 8168 webserver.cc:460] Webserver started at http://127.23.115.1:35971/ using document root <none> and password file <none>
I20260620 18:42:39.514506 8168 fs_manager.cc:359] Metadata directory not provided
I20260620 18:42:39.514623 8168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:42:39.520372 8168 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 18:42:39.523432 8223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:39.525019 8168 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 18:42:39.525202 8168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-0/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-0/wal
uuid: "86629c1da49b4263ae52078ef6a60192"
format_stamp: "Formatted at 2026-06-20 18:41:53 on dist-test-slave-d645"
server_key: "a45a2eff8f0bc3ba6e9dc818944ad5ae"
server_key_iv: "7ae3c4659b337945e16c951f830ad5b9"
server_key_version: "encryptionkey@0"
I20260620 18:42:39.526352 8168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:42:39.557962 8168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:42:39.562888 8168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:42:39.563175 8168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:42:39.564153 8168 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:42:39.565541 8168 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:42:39.565629 8168 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:39.565711 8168 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:42:39.565768 8168 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:39.600061 8168 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.1:46605
I20260620 18:42:39.600135 8335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.1:46605 every 8 connection(s)
I20260620 18:42:39.601464 8168 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-0/data/info.pb
I20260620 18:42:39.603271 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 8168
I20260620 18:42:39.603881 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.115.2:35821
--local_ip_for_outbound_sockets=127.23.115.2
--tserver_master_addrs=127.23.115.62:44779
--webserver_port=37537
--webserver_interface=127.23.115.2
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.2.keytab
--principal=kudu/127.23.115.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:41715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:42:39.611624 8338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:39.604485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:44779 (local address 127.23.115.1:49117)
0620 18:42:39.610324 (+ 5839us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:39.610638 (+ 314us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:44779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5555,"thread_start_us":5364,"threads_started":1}
W20260620 18:42:39.612457 8336 heartbeater.cc:644] Failed to heartbeat to 127.23.115.62:44779 (0 consecutive failures): Network error: Failed to ping master at 127.23.115.62:44779: Client connection negotiation failed: client connection to 127.23.115.62:44779: connect: Connection refused (error 111)
I20260620 18:42:39.613369 8338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:39.613010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:44779 (local address 127.23.115.1:59813)
0620 18:42:39.613162 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:39.613244 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:44779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260620 18:42:39.614265 8338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:39.613902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:44779 (local address 127.23.115.1:45339)
0620 18:42:39.614053 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:39.614140 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:44779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260620 18:42:39.614501 8336 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 18:42:39.939347 8339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:42:39.939636 8339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:42:39.939735 8339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:42:39.947562 8339 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:42:39.947707 8339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:42:39.947839 8339 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.115.2
I20260620 18:42:39.958841 8339 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:41715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.2.keytab
--principal=kudu/127.23.115.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.115.2:35821
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.115.2
--webserver_port=37537
--tserver_master_addrs=127.23.115.62:44779
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.115.2
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:42:39.960304 8339 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:42:39.961596 8339 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:42:39.976574 8344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:39.976644 8345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:39.976549 8347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:42:39.983355 8339 server_base.cc:1034] running on GCE node
I20260620 18:42:39.984148 8339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:42:39.985293 8339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:42:39.986548 8339 hybrid_clock.cc:648] HybridClock initialized: now 1781980959986495 us; error 43 us; skew 500 ppm
I20260620 18:42:39.986903 8339 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:42:39 dist-test-slave-d645 krb5kdc[7020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980959, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:42:39.990221 8339 init.cc:377] Logged in from keytab as kudu/127.23.115.2@KRBTEST.COM (short username kudu)
I20260620 18:42:39.992028 8339 webserver.cc:460] Webserver started at http://127.23.115.2:37537/ using document root <none> and password file <none>
I20260620 18:42:39.992753 8339 fs_manager.cc:359] Metadata directory not provided
I20260620 18:42:39.992870 8339 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:42:39.998430 8339 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260620 18:42:40.001703 8354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:40.003182 8339 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:42:40.003351 8339 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-1/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-1/wal
uuid: "5de3fa128ee24886be30c46b379343ba"
format_stamp: "Formatted at 2026-06-20 18:41:54 on dist-test-slave-d645"
server_key: "de72182d925d52935bdba1272e6b5600"
server_key_iv: "b67d0c1f4c2e2d24dfdc13b6c80a1dd5"
server_key_version: "encryptionkey@0"
I20260620 18:42:40.004349 8339 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:42:40.023260 8339 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:42:40.027876 8339 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:42:40.028175 8339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:42:40.029255 8339 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:42:40.030838 8339 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:42:40.030966 8339 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:40.031090 8339 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:42:40.031170 8339 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:40.065259 8339 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.2:35821
I20260620 18:42:40.065331 8466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.2:35821 every 8 connection(s)
I20260620 18:42:40.066679 8339 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-1/data/info.pb
I20260620 18:42:40.071369 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 8339
I20260620 18:42:40.072144 8469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:40.069805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:44779 (local address 127.23.115.2:52851)
0620 18:42:40.070631 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:40.070897 (+ 266us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:44779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":366,"threads_started":1}
I20260620 18:42:40.072142 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.115.3:38983
--local_ip_for_outbound_sockets=127.23.115.3
--tserver_master_addrs=127.23.115.62:44779
--webserver_port=34405
--webserver_interface=127.23.115.3
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.3.keytab
--principal=kudu/127.23.115.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:41715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:42:40.073351 8467 heartbeater.cc:644] Failed to heartbeat to 127.23.115.62:44779 (0 consecutive failures): Network error: Failed to ping master at 127.23.115.62:44779: Client connection negotiation failed: client connection to 127.23.115.62:44779: connect: Connection refused (error 111)
I20260620 18:42:40.074889 8469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:40.074153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:44779 (local address 127.23.115.2:38895)
0620 18:42:40.074452 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:40.074603 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:44779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260620 18:42:40.076128 8469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:40.075662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:44779 (local address 127.23.115.2:44773)
0620 18:42:40.075830 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:40.075940 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:44779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 18:42:40.076411 8467 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 18:42:40.397924 8470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:42:40.398203 8470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:42:40.398321 8470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:42:40.405721 8470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:42:40.405857 8470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:42:40.406032 8470 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.115.3
I20260620 18:42:40.416484 8470 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:41715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.3.keytab
--principal=kudu/127.23.115.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.115.3:38983
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.115.3
--webserver_port=34405
--tserver_master_addrs=127.23.115.62:44779
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.115.3
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:42:40.417855 8470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:42:40.419215 8470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:42:40.432549 8475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:40.432511 8476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:40.432480 8478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:42:40.433584 8470 server_base.cc:1034] running on GCE node
I20260620 18:42:40.434230 8470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:42:40.435338 8470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:42:40.436573 8470 hybrid_clock.cc:648] HybridClock initialized: now 1781980960436527 us; error 39 us; skew 500 ppm
I20260620 18:42:40.436940 8470 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:42:40 dist-test-slave-d645 krb5kdc[7020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980960, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:42:40.439986 8470 init.cc:377] Logged in from keytab as kudu/127.23.115.3@KRBTEST.COM (short username kudu)
I20260620 18:42:40.441438 8470 webserver.cc:460] Webserver started at http://127.23.115.3:34405/ using document root <none> and password file <none>
I20260620 18:42:40.442095 8470 fs_manager.cc:359] Metadata directory not provided
I20260620 18:42:40.442206 8470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:42:40.447573 8470 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:42:40.450332 8485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:40.451761 8470 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:42:40.451917 8470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-2/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-2/wal
uuid: "8205483c83dc4727a92ef655e516de62"
format_stamp: "Formatted at 2026-06-20 18:41:54 on dist-test-slave-d645"
server_key: "fb6c68aa3f0f57925a759e5920ef2f3c"
server_key_iv: "417abd0bfa840c397f81c408baafc9c1"
server_key_version: "encryptionkey@0"
I20260620 18:42:40.452881 8470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:42:40.471423 8470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:42:40.475975 8470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:42:40.476234 8470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:42:40.477217 8470 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:42:40.478684 8470 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:42:40.478788 8470 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:40.478962 8470 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:42:40.479038 8470 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260620 18:42:40.510813 8470 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.3:38983
I20260620 18:42:40.510885 8597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.3:38983 every 8 connection(s)
I20260620 18:42:40.512216 8470 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980676253068-24012-0/minicluster-data/ts-2/data/info.pb
I20260620 18:42:40.516973 8600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:40.515027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:44779 (local address 127.23.115.3:33693)
0620 18:42:40.515883 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:40.516112 (+ 229us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:44779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":313,"threads_started":1}
I20260620 18:42:40.517473 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 8470
W20260620 18:42:40.517724 8598 heartbeater.cc:644] Failed to heartbeat to 127.23.115.62:44779 (0 consecutive failures): Network error: Failed to ping master at 127.23.115.62:44779: Client connection negotiation failed: client connection to 127.23.115.62:44779: connect: Connection refused (error 111)
I20260620 18:42:40.519124 8600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:40.518348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:44779 (local address 127.23.115.3:40257)
0620 18:42:40.518688 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:40.518864 (+ 176us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:44779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260620 18:42:40.520291 8600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:40.519852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:44779 (local address 127.23.115.3:56515)
0620 18:42:40.520046 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:40.520144 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:44779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260620 18:42:40.520534 8598 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 18:42:40.616061 8602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:40.615179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:44779 (local address 127.23.115.1:35251)
0620 18:42:40.615703 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:40.615802 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:44779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":233,"threads_started":1}
I20260620 18:42:41.078195 8603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:41.076987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:44779 (local address 127.23.115.2:45187)
0620 18:42:41.077794 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:41.077957 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:44779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":329,"threads_started":1}
I20260620 18:42:41.522003 8604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:41.521108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:44779 (local address 127.23.115.3:57913)
0620 18:42:41.521558 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:41.521668 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:44779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":204,"threads_started":1}
I20260620 18:42:41.617760 8605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:41.617028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:44779 (local address 127.23.115.1:39999)
0620 18:42:41.617508 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:41.617590 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:44779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":248,"threads_started":1}
I20260620 18:42:42.079991 8606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:42.079261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:44779 (local address 127.23.115.2:34141)
0620 18:42:42.079737 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:42.079822 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:44779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":239,"threads_started":1}
I20260620 18:42:42.523494 8607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:42.522813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:44779 (local address 127.23.115.3:54091)
0620 18:42:42.523251 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:42.523327 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:44779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":226,"threads_started":1}
I20260620 18:42:42.619585 8608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:42.618828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:44779 (local address 127.23.115.1:33559)
0620 18:42:42.619325 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:42.619417 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:44779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":288,"threads_started":1}
I20260620 18:42:43.081780 8609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:43.080978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:44779 (local address 127.23.115.2:60029)
0620 18:42:43.081481 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:43.081577 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:44779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":281,"threads_started":1}
I20260620 18:42:43.525493 8610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:43.524606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:44779 (local address 127.23.115.3:41737)
0620 18:42:43.525205 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:43.525304 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:44779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":331,"threads_started":1}
I20260620 18:42:43.621251 8611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:43.620513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:44779 (local address 127.23.115.1:50269)
0620 18:42:43.620987 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:43.621069 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:44779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":234,"threads_started":1}
I20260620 18:42:44.083560 8612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:44.082771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:44779 (local address 127.23.115.2:37129)
0620 18:42:44.083297 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:44.083381 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:44779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":313,"threads_started":1}
I20260620 18:42:44.527567 8613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:44.526447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:44779 (local address 127.23.115.3:42287)
0620 18:42:44.527262 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:44.527360 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:44779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":336,"threads_started":1}
I20260620 18:42:44.623025 8614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:44.622195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:44779 (local address 127.23.115.1:44269)
0620 18:42:44.622704 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:44.622810 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:44779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":247,"threads_started":1}
I20260620 18:42:45.085829 8615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:45.085013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:44779 (local address 127.23.115.2:54241)
0620 18:42:45.085533 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:45.085629 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:44779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":277,"threads_started":1}
I20260620 18:42:45.529464 8616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:45.528612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:44779 (local address 127.23.115.3:46175)
0620 18:42:45.529203 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:45.529288 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:44779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":273,"threads_started":1}
I20260620 18:42:45.624862 8617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:45.623974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:44779 (local address 127.23.115.1:51347)
0620 18:42:45.624507 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:45.624630 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:44779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":263,"threads_started":1}
I20260620 18:42:46.087786 8618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:46.086895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:44779 (local address 127.23.115.2:36707)
0620 18:42:46.087439 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:46.087579 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:44779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":243,"threads_started":1}
I20260620 18:42:46.531147 8619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:46.530408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:44779 (local address 127.23.115.3:54101)
0620 18:42:46.530824 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:46.530911 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:44779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":214,"threads_started":1}
I20260620 18:42:46.627000 8620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:46.626096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:44779 (local address 127.23.115.1:38269)
0620 18:42:46.626635 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:46.626730 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:44779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":233,"threads_started":1}
I20260620 18:42:47.089617 8621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:47.088808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:44779 (local address 127.23.115.2:37547)
0620 18:42:47.089312 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:47.089397 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:44779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":228,"threads_started":1}
I20260620 18:42:47.533063 8622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:47.532238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:44779 (local address 127.23.115.3:33037)
0620 18:42:47.532751 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:47.532836 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:44779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":208,"threads_started":1}
I20260620 18:42:47.628752 8623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:47.628074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:44779 (local address 127.23.115.1:58003)
0620 18:42:47.628516 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:47.628594 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:44779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":230,"threads_started":1}
I20260620 18:42:48.091357 8624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:48.090537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:44779 (local address 127.23.115.2:57675)
0620 18:42:48.091077 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:48.091172 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:44779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":243,"threads_started":1}
I20260620 18:42:48.534832 8625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:48.534104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:44779 (local address 127.23.115.3:42697)
0620 18:42:48.534550 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:48.534660 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:44779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":234,"threads_started":1}
I20260620 18:42:48.630676 8626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:48.629772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:44779 (local address 127.23.115.1:43345)
0620 18:42:48.630330 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:48.630494 (+ 164us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:44779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":277,"threads_started":1}
I20260620 18:42:49.093048 8627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:49.092297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:44779 (local address 127.23.115.2:38871)
0620 18:42:49.092787 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:49.092866 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:44779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":255,"threads_started":1}
I20260620 18:42:49.536538 8628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:49.535680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:44779 (local address 127.23.115.3:45795)
0620 18:42:49.536246 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:49.536336 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:44779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":276,"threads_started":1}
I20260620 18:42:49.632978 8629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:49.631873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:44779 (local address 127.23.115.1:57577)
0620 18:42:49.632672 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:49.632784 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:44779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":637,"thread_start_us":375,"threads_started":1}
I20260620 18:42:50.094784 8630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:50.094016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:44779 (local address 127.23.115.2:43721)
0620 18:42:50.094505 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:50.094617 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:44779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":265,"threads_started":1}
I20260620 18:42:50.538340 8631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:50.537580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:44779 (local address 127.23.115.3:42399)
0620 18:42:50.538091 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:50.538176 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:44779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":278,"threads_started":1}
I20260620 18:42:50.634903 8632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:50.634005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:44779 (local address 127.23.115.1:51451)
0620 18:42:50.634595 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:50.634703 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:44779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":265,"threads_started":1}
I20260620 18:42:51.096606 8633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:51.095743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:44779 (local address 127.23.115.2:56829)
0620 18:42:51.096313 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:51.096409 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:44779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":335,"threads_started":1}
I20260620 18:42:51.540189 8634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:51.539435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:44779 (local address 127.23.115.3:58071)
0620 18:42:51.539911 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:51.540003 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:44779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":254,"threads_started":1}
I20260620 18:42:51.636687 8635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:51.635922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:44779 (local address 127.23.115.1:53247)
0620 18:42:51.636401 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:51.636518 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:44779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":253,"threads_started":1}
I20260620 18:42:52.098456 8636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:52.097767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:44779 (local address 127.23.115.2:51985)
0620 18:42:52.098217 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:52.098299 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:44779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":232,"threads_started":1}
I20260620 18:42:52.542124 8637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:52.541330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:44779 (local address 127.23.115.3:44875)
0620 18:42:52.541843 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:52.541932 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:44779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":266,"threads_started":1}
I20260620 18:42:52.638859 8638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:52.637938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:44779 (local address 127.23.115.1:36085)
0620 18:42:52.638541 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:52.638669 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:44779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":263,"threads_started":1}
I20260620 18:42:53.100220 8639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:53.099424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:44779 (local address 127.23.115.2:56393)
0620 18:42:53.099928 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:53.100033 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:44779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":248,"threads_started":1}
I20260620 18:42:53.544550 8640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:53.543318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:44779 (local address 127.23.115.3:35537)
0620 18:42:53.544192 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:53.544317 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:44779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":632,"thread_start_us":440,"threads_started":1}
I20260620 18:42:53.640538 8641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:53.639813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:44779 (local address 127.23.115.1:53989)
0620 18:42:53.640260 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:53.640358 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:44779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":218,"threads_started":1}
I20260620 18:42:54.102279 8642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:54.101520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:44779 (local address 127.23.115.2:39337)
0620 18:42:54.102005 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:54.102088 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:44779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":253,"threads_started":1}
I20260620 18:42:54.546615 8643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:54.545838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:44779 (local address 127.23.115.3:38187)
0620 18:42:54.546326 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:54.546413 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:44779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":272,"threads_started":1}
I20260620 18:42:54.642493 8644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:54.641572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:44779 (local address 127.23.115.1:56145)
0620 18:42:54.642065 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:54.642268 (+ 203us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:44779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":295,"threads_started":1}
I20260620 18:42:55.104539 8645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:55.103691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:44779 (local address 127.23.115.2:49471)
0620 18:42:55.104253 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:55.104351 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:44779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":256,"threads_started":1}
I20260620 18:42:55.523098 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 8168
I20260620 18:42:55.546394 8646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:55.524452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38948 (local address 127.23.115.1:46605)
0620 18:42:55.525133 (+ 681us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:55.525148 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:55.525187 (+ 39us) server_negotiation.cc:407] Connection header received
0620 18:42:55.526805 (+ 1618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:55.526815 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:55.527305 (+ 490us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:55.527648 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:55.529316 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:55.530617 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:55.531982 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:55.532716 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:55.532906 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:55.540149 (+ 7243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:55.540158 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:55.540176 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:55.540219 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:55.543138 (+ 2919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:55.543861 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:55.543868 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:55.543875 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:55.544052 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:55.544400 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:55.544404 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:55.544406 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:55.544919 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:55.545234 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:55.545699 (+ 465us) server_negotiation.cc:299] Negotiation successful
0620 18:42:55.546118 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":378,"threads_started":1}
I20260620 18:42:55.548635 8331 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:42:55.548830 8647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:55.547933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:44779 (local address 127.23.115.3:33481)
0620 18:42:55.548477 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:55.548614 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.62:44779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":268,"threads_started":1}
I20260620 18:42:55.632961 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 8168
I20260620 18:42:55.648526 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 8339
I20260620 18:42:55.667040 8649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:55.649540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42192 (local address 127.23.115.2:35821)
0620 18:42:55.650043 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:55.650057 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:55.650092 (+ 35us) server_negotiation.cc:407] Connection header received
0620 18:42:55.651572 (+ 1480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:55.651582 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:55.652000 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:55.652269 (+ 269us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:55.653677 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:55.654912 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:55.656333 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:55.657438 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:55.657956 (+ 518us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:55.660382 (+ 2426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:55.660392 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:55.660409 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:55.660463 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:55.663430 (+ 2967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:55.664115 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:55.664123 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:55.664129 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:55.664297 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:55.665064 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:55.665068 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:55.665070 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:55.665637 (+ 567us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:55.665879 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:55.666291 (+ 412us) server_negotiation.cc:299] Negotiation successful
0620 18:42:55.666702 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":226,"threads_started":1}
I20260620 18:42:55.668424 8462 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:42:55.772579 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 8339
I20260620 18:42:55.787786 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 8470
I20260620 18:42:55.806907 8651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:55.788740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35766 (local address 127.23.115.3:38983)
0620 18:42:55.789324 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:55.789340 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:55.789383 (+ 43us) server_negotiation.cc:407] Connection header received
0620 18:42:55.791512 (+ 2129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:55.791529 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:55.792208 (+ 679us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:55.792515 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:55.793904 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:55.795278 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:55.796522 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:55.797237 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:55.797377 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:55.800558 (+ 3181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:55.800571 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:55.800587 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:55.800626 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:55.803781 (+ 3155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:55.804450 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:55.804457 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:55.804463 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:55.804625 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:55.805050 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:55.805053 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:55.805056 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:55.805582 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:55.805858 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:55.806274 (+ 416us) server_negotiation.cc:299] Negotiation successful
0620 18:42:55.806606 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":252,"threads_started":1}
I20260620 18:42:55.808297 8593 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:42:55.891166 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 8470
2026-06-20T18:42:55Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64615 ms)
[----------] 30 tests from SecurityITest (299601 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980676253068-24012-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:42:55 dist-test-slave-d645 krb5kdc[8655](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:42:55 dist-test-slave-d645 krb5kdc[8655](info): set up 2 sockets
Jun 20 18:42:55 dist-test-slave-d645 krb5kdc[8655](info): commencing operation
krb5kdc: starting...
W20260620 18:42:56.931061 24012 mini_kdc.cc:121] Time spent starting KDC: real 1.003s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980676253068-24012-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980676253068-24012-0/krb5kdc/test-user.keytab.
Jun 20 18:42:57 dist-test-slave-d645 krb5kdc[8655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980977, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:42:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:42:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.115.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.115.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.115.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab.
Entry for principal kudu/127.23.115.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab.
I20260620 18:42:57.176524 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980676253068-24012-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980676253068-24012-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980676253068-24012-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.115.62:40995
--webserver_interface=127.23.115.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab
--principal=kudu/127.23.115.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:46561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.115.62:40995
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:42:57.516877 8668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:42:57.517230 8668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:42:57.517371 8668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:42:57.524745 8668 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:42:57.524865 8668 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:42:57.524945 8668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:42:57.525030 8668 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:42:57.525094 8668 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:42:57.535068 8668 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:46561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980676253068-24012-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980676253068-24012-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.115.62:40995
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab
--principal=kudu/127.23.115.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.115.62:40995
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.115.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980676253068-24012-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:42:57.536415 8668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:42:57.537672 8668 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:42:57.549271 8683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:57.549324 8674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:57.549350 8675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:42:57.549952 8668 server_base.cc:1034] running on GCE node
I20260620 18:42:57.550717 8668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:42:57.552134 8668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:42:57.553339 8668 hybrid_clock.cc:648] HybridClock initialized: now 1781980977553296 us; error 36 us; skew 500 ppm
I20260620 18:42:57.553669 8668 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:42:57 dist-test-slave-d645 krb5kdc[8655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980977, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:42:57.556661 8668 init.cc:377] Logged in from keytab as kudu/127.23.115.62@KRBTEST.COM (short username kudu)
I20260620 18:42:57.558408 8668 webserver.cc:460] Webserver started at http://127.23.115.62:42679/ using document root <none> and password file <none>
I20260620 18:42:57.559126 8668 fs_manager.cc:359] Metadata directory not provided
I20260620 18:42:57.559239 8668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:42:57.559558 8668 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:42:57.562503 8668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980676253068-24012-0/minicluster-data/master-0/data/instance:
uuid: "f1002ae35eca41f8ae3127886d3c5b6b"
format_stamp: "Formatted at 2026-06-20 18:42:57 on dist-test-slave-d645"
server_key: "98f9a8c5d6bbfcc8ebfe0aa4499d6a11"
server_key_iv: "31b6861c26d59c25b2c3605c4e891507"
server_key_version: "encryptionkey@0"
I20260620 18:42:57.563258 8668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980676253068-24012-0/minicluster-data/master-0/wal/instance:
uuid: "f1002ae35eca41f8ae3127886d3c5b6b"
format_stamp: "Formatted at 2026-06-20 18:42:57 on dist-test-slave-d645"
server_key: "98f9a8c5d6bbfcc8ebfe0aa4499d6a11"
server_key_iv: "31b6861c26d59c25b2c3605c4e891507"
server_key_version: "encryptionkey@0"
I20260620 18:42:57.568286 8668 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 18:42:57.571643 8690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:57.573011 8668 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:42:57.573158 8668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980676253068-24012-0/minicluster-data/master-0/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980676253068-24012-0/minicluster-data/master-0/wal
uuid: "f1002ae35eca41f8ae3127886d3c5b6b"
format_stamp: "Formatted at 2026-06-20 18:42:57 on dist-test-slave-d645"
server_key: "98f9a8c5d6bbfcc8ebfe0aa4499d6a11"
server_key_iv: "31b6861c26d59c25b2c3605c4e891507"
server_key_version: "encryptionkey@0"
I20260620 18:42:57.573340 8668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980676253068-24012-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980676253068-24012-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980676253068-24012-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:42:57.590716 8668 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:42:57.595055 8668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:42:57.595333 8668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:42:57.613972 8668 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.62:40995
I20260620 18:42:57.613981 8741 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.62:40995 every 8 connection(s)
I20260620 18:42:57.615347 8668 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
I20260620 18:42:57.619062 8742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:42:57.620963 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 8668
I20260620 18:42:57.621191 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980676253068-24012-0/minicluster-data/master-0/wal/instance
I20260620 18:42:57.621966 24012 external_mini_cluster.cc:1428] Setting key b2d382effc91d6e2c1d4208e63b7403b
I20260620 18:42:57.631916 8742 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1002ae35eca41f8ae3127886d3c5b6b: Bootstrap starting.
I20260620 18:42:57.636963 8742 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f1002ae35eca41f8ae3127886d3c5b6b: Neither blocks nor log segments found. Creating new log.
I20260620 18:42:57.638836 8742 log.cc:824] T 00000000000000000000000000000000 P f1002ae35eca41f8ae3127886d3c5b6b: Log is configured to *not* fsync() on all Append() calls
Jun 20 18:42:57 dist-test-slave-d645 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980977, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.115.62@KRBTEST.COM
I20260620 18:42:57.642266 8742 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1002ae35eca41f8ae3127886d3c5b6b: No bootstrap required, opened a new log
I20260620 18:42:57.647985 8745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:57.624166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35394 (local address 127.23.115.62:40995)
0620 18:42:57.624942 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:57.624954 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:57.624997 (+ 43us) server_negotiation.cc:407] Connection header received
0620 18:42:57.626691 (+ 1694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:57.626713 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:57.627250 (+ 537us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:57.627560 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:57.629417 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:57.630791 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:57.632880 (+ 2089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:57.633909 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:57.634131 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:57.640776 (+ 6645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:57.640786 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:57.640803 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:57.640844 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:57.643858 (+ 3014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:57.644560 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:57.644567 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:57.644574 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:57.644755 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:57.645123 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:57.645138 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:57.645140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:57.645731 (+ 591us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:57.646051 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:57.646451 (+ 400us) server_negotiation.cc:299] Negotiation successful
0620 18:42:57.646903 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":330,"threads_started":1}
I20260620 18:42:57.651373 8742 raft_consensus.cc:348] T 00000000000000000000000000000000 P f1002ae35eca41f8ae3127886d3c5b6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1002ae35eca41f8ae3127886d3c5b6b" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 40995 } }
I20260620 18:42:57.651767 8742 raft_consensus.cc:374] T 00000000000000000000000000000000 P f1002ae35eca41f8ae3127886d3c5b6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:42:57.651863 8742 raft_consensus.cc:729] T 00000000000000000000000000000000 P f1002ae35eca41f8ae3127886d3c5b6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1002ae35eca41f8ae3127886d3c5b6b, State: Initialized, Role: FOLLOWER
I20260620 18:42:57.652557 8742 consensus_queue.cc:260] T 00000000000000000000000000000000 P f1002ae35eca41f8ae3127886d3c5b6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1002ae35eca41f8ae3127886d3c5b6b" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 40995 } }
I20260620 18:42:57.652799 8742 raft_consensus.cc:388] T 00000000000000000000000000000000 P f1002ae35eca41f8ae3127886d3c5b6b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:42:57.652923 8742 raft_consensus.cc:482] T 00000000000000000000000000000000 P f1002ae35eca41f8ae3127886d3c5b6b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:42:57.653069 8742 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f1002ae35eca41f8ae3127886d3c5b6b [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:42:57.655279 8742 raft_consensus.cc:504] T 00000000000000000000000000000000 P f1002ae35eca41f8ae3127886d3c5b6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1002ae35eca41f8ae3127886d3c5b6b" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 40995 } }
I20260620 18:42:57.655799 8742 leader_election.cc:302] T 00000000000000000000000000000000 P f1002ae35eca41f8ae3127886d3c5b6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f1002ae35eca41f8ae3127886d3c5b6b; no voters:
I20260620 18:42:57.656348 8742 leader_election.cc:288] T 00000000000000000000000000000000 P f1002ae35eca41f8ae3127886d3c5b6b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:42:57.656481 8747 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f1002ae35eca41f8ae3127886d3c5b6b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:42:57.657781 8747 raft_consensus.cc:686] T 00000000000000000000000000000000 P f1002ae35eca41f8ae3127886d3c5b6b [term 1 LEADER]: Becoming Leader. State: Replica: f1002ae35eca41f8ae3127886d3c5b6b, State: Running, Role: LEADER
I20260620 18:42:57.658331 8747 consensus_queue.cc:237] T 00000000000000000000000000000000 P f1002ae35eca41f8ae3127886d3c5b6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1002ae35eca41f8ae3127886d3c5b6b" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 40995 } }
I20260620 18:42:57.658565 8742 sys_catalog.cc:564] T 00000000000000000000000000000000 P f1002ae35eca41f8ae3127886d3c5b6b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:42:57.661088 8748 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1002ae35eca41f8ae3127886d3c5b6b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f1002ae35eca41f8ae3127886d3c5b6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1002ae35eca41f8ae3127886d3c5b6b" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 40995 } } }
I20260620 18:42:57.661114 8749 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1002ae35eca41f8ae3127886d3c5b6b [sys.catalog]: SysCatalogTable state changed. Reason: New leader f1002ae35eca41f8ae3127886d3c5b6b. Latest consensus state: current_term: 1 leader_uuid: "f1002ae35eca41f8ae3127886d3c5b6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1002ae35eca41f8ae3127886d3c5b6b" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 40995 } } }
I20260620 18:42:57.661633 8749 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1002ae35eca41f8ae3127886d3c5b6b [sys.catalog]: This master's current role is: LEADER
I20260620 18:42:57.661633 8748 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1002ae35eca41f8ae3127886d3c5b6b [sys.catalog]: This master's current role is: LEADER
I20260620 18:42:57.663349 8755 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:42:57.668761 8755 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:42:57.678694 8755 catalog_manager.cc:1347] Generated new cluster ID: bd13d36c2c3e4161b3f39048867aa764
I20260620 18:42:57.678812 8755 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:42:57.703954 8755 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:42:57.705330 8755 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:42:57.720896 8755 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f1002ae35eca41f8ae3127886d3c5b6b: Generated new TSK 0
I20260620 18:42:57.721959 8755 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.115.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.115.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.115.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.1.keytab.
Entry for principal kudu/127.23.115.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.1.keytab.
I20260620 18:42:57.786612 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980676253068-24012-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980676253068-24012-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980676253068-24012-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980676253068-24012-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.115.1:0
--local_ip_for_outbound_sockets=127.23.115.1
--webserver_interface=127.23.115.1
--webserver_port=0
--tserver_master_addrs=127.23.115.62:40995
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.1.keytab
--principal=kudu/127.23.115.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:46561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:42:58.119310 8768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:42:58.119588 8768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:42:58.119709 8768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:42:58.127628 8768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:42:58.127749 8768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:42:58.127905 8768 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.115.1
I20260620 18:42:58.138330 8768 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:46561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980676253068-24012-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980676253068-24012-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.1.keytab
--principal=kudu/127.23.115.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.115.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980676253068-24012-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.115.1
--webserver_port=0
--tserver_master_addrs=127.23.115.62:40995
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.115.1
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980676253068-24012-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:42:58.139905 8768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:42:58.141261 8768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:42:58.154443 8773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:58.154469 8776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:58.154368 8774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:42:58.155464 8768 server_base.cc:1034] running on GCE node
I20260620 18:42:58.156033 8768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:42:58.157006 8768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:42:58.158244 8768 hybrid_clock.cc:648] HybridClock initialized: now 1781980978158184 us; error 50 us; skew 500 ppm
I20260620 18:42:58.158596 8768 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:42:58 dist-test-slave-d645 krb5kdc[8655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980978, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:42:58.161780 8768 init.cc:377] Logged in from keytab as kudu/127.23.115.1@KRBTEST.COM (short username kudu)
I20260620 18:42:58.163659 8768 webserver.cc:460] Webserver started at http://127.23.115.1:37177/ using document root <none> and password file <none>
I20260620 18:42:58.164409 8768 fs_manager.cc:359] Metadata directory not provided
I20260620 18:42:58.164547 8768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:42:58.164872 8768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:42:58.168190 8768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980676253068-24012-0/minicluster-data/ts-0/data/instance:
uuid: "c848eed8b6d44403a46a73dbc59cb289"
format_stamp: "Formatted at 2026-06-20 18:42:58 on dist-test-slave-d645"
server_key: "54b8d91fae5334c05b28ada3b1c82749"
server_key_iv: "99765e440a7079f7dc651f9f76716656"
server_key_version: "encryptionkey@0"
I20260620 18:42:58.169157 8768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980676253068-24012-0/minicluster-data/ts-0/wal/instance:
uuid: "c848eed8b6d44403a46a73dbc59cb289"
format_stamp: "Formatted at 2026-06-20 18:42:58 on dist-test-slave-d645"
server_key: "54b8d91fae5334c05b28ada3b1c82749"
server_key_iv: "99765e440a7079f7dc651f9f76716656"
server_key_version: "encryptionkey@0"
I20260620 18:42:58.174511 8768 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260620 18:42:58.178296 8783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:58.179838 8768 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:42:58.180040 8768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980676253068-24012-0/minicluster-data/ts-0/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980676253068-24012-0/minicluster-data/ts-0/wal
uuid: "c848eed8b6d44403a46a73dbc59cb289"
format_stamp: "Formatted at 2026-06-20 18:42:58 on dist-test-slave-d645"
server_key: "54b8d91fae5334c05b28ada3b1c82749"
server_key_iv: "99765e440a7079f7dc651f9f76716656"
server_key_version: "encryptionkey@0"
I20260620 18:42:58.180255 8768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980676253068-24012-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980676253068-24012-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980676253068-24012-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:42:58.192799 8768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:42:58.197209 8768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:42:58.197551 8768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:42:58.198570 8768 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:42:58.200116 8768 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:42:58.200202 8768 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:58.200335 8768 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:42:58.200394 8768 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:58.233345 8768 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.1:43789
I20260620 18:42:58.233385 8895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.1:43789 every 8 connection(s)
I20260620 18:42:58.234596 8768 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980676253068-24012-0/minicluster-data/ts-0/data/info.pb
I20260620 18:42:58.240558 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 8768
I20260620 18:42:58.240809 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980676253068-24012-0/minicluster-data/ts-0/wal/instance
I20260620 18:42:58.241607 24012 external_mini_cluster.cc:1428] Setting key 7e92f33584791eea710287899be20d63
Jun 20 18:42:58 dist-test-slave-d645 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980978, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.1@KRBTEST.COM for kudu/127.23.115.62@KRBTEST.COM
I20260620 18:42:58.255527 8898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:58.237261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.1:58959 (local address 127.23.115.62:40995)
0620 18:42:58.237733 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:58.237740 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:58.238696 (+ 956us) server_negotiation.cc:407] Connection header received
0620 18:42:58.240585 (+ 1889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:58.240592 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:58.240691 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:58.240836 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:58.242470 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:58.243358 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:58.244975 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:58.245522 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:58.245650 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:58.249019 (+ 3369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:58.249025 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:58.249029 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:58.249066 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:58.251920 (+ 2854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:58.252846 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:58.252852 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:58.252854 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:58.252982 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:58.253470 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:58.253474 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:58.253477 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:58.253871 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:58.254000 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:58.255049 (+ 1049us) server_negotiation.cc:299] Negotiation successful
0620 18:42:58.255249 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":206,"threads_started":1}
I20260620 18:42:58.256899 8899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:58.237818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:40995 (local address 127.23.115.1:58959)
0620 18:42:58.238530 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:58.238566 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:58.240272 (+ 1706us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:58.241057 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:58.241065 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:58.241722 (+ 657us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:42:58.242312 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:58.242330 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:58.243515 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:58.243520 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:58.244803 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:58.244816 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:58.245764 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:58.245769 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:58.245968 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:58.246607 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:42:58.246636 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:42:58.248788 (+ 2152us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:42:58.252079 (+ 3291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:58.252088 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:58.252114 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:58.252701 (+ 587us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:58.253094 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:58.253104 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:58.253108 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:58.253340 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:58.254039 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:42:58.254050 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:42:58.254509 (+ 459us) client_negotiation.cc:769] Sending connection context
0620 18:42:58.254867 (+ 358us) client_negotiation.cc:240] Negotiation successful
0620 18:42:58.255460 (+ 593us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":292,"threads_started":1}
I20260620 18:42:58.259207 8896 heartbeater.cc:347] Connected to a master server at 127.23.115.62:40995
I20260620 18:42:58.259567 8896 heartbeater.cc:464] Registering TS with master...
I20260620 18:42:58.260329 8896 heartbeater.cc:511] Master 127.23.115.62:40995 requested a full tablet report, sending...
I20260620 18:42:58.263080 8707 ts_manager.cc:194] Registered new tserver with Master: c848eed8b6d44403a46a73dbc59cb289 (127.23.115.1:43789)
I20260620 18:42:58.265198 8707 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.115.1@KRBTEST.COM'} at 127.23.115.1:58959
WARNING: no policy specified for kudu/127.23.115.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.115.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.115.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.2.keytab.
Entry for principal kudu/127.23.115.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.2.keytab.
I20260620 18:42:58.299510 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980676253068-24012-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980676253068-24012-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980676253068-24012-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980676253068-24012-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.115.2:0
--local_ip_for_outbound_sockets=127.23.115.2
--webserver_interface=127.23.115.2
--webserver_port=0
--tserver_master_addrs=127.23.115.62:40995
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.2.keytab
--principal=kudu/127.23.115.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:46561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:42:58.626417 8902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:42:58.626688 8902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:42:58.626816 8902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:42:58.634572 8902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:42:58.634712 8902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:42:58.634876 8902 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.115.2
I20260620 18:42:58.645226 8902 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:46561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980676253068-24012-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980676253068-24012-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.2.keytab
--principal=kudu/127.23.115.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.115.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980676253068-24012-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.115.2
--webserver_port=0
--tserver_master_addrs=127.23.115.62:40995
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.115.2
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980676253068-24012-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:42:58.646576 8902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:42:58.648088 8902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:42:58.661620 8907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:58.661724 8910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:58.661656 8908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:42:58.662549 8902 server_base.cc:1034] running on GCE node
I20260620 18:42:58.663259 8902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:42:58.664240 8902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:42:58.665468 8902 hybrid_clock.cc:648] HybridClock initialized: now 1781980978665416 us; error 42 us; skew 500 ppm
I20260620 18:42:58.665812 8902 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:42:58 dist-test-slave-d645 krb5kdc[8655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980978, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:42:58.668953 8902 init.cc:377] Logged in from keytab as kudu/127.23.115.2@KRBTEST.COM (short username kudu)
I20260620 18:42:58.670477 8902 webserver.cc:460] Webserver started at http://127.23.115.2:45519/ using document root <none> and password file <none>
I20260620 18:42:58.671387 8902 fs_manager.cc:359] Metadata directory not provided
I20260620 18:42:58.671523 8902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:42:58.671844 8902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:42:58.674911 8902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980676253068-24012-0/minicluster-data/ts-1/data/instance:
uuid: "924265114bec4e2c9b822fdd707050b0"
format_stamp: "Formatted at 2026-06-20 18:42:58 on dist-test-slave-d645"
server_key: "10f5ca7f35f936e7b6fe13dfe5a33230"
server_key_iv: "ee3330e15e274c1e2a3a4ec1711770c1"
server_key_version: "encryptionkey@0"
I20260620 18:42:58.675773 8902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980676253068-24012-0/minicluster-data/ts-1/wal/instance:
uuid: "924265114bec4e2c9b822fdd707050b0"
format_stamp: "Formatted at 2026-06-20 18:42:58 on dist-test-slave-d645"
server_key: "10f5ca7f35f936e7b6fe13dfe5a33230"
server_key_iv: "ee3330e15e274c1e2a3a4ec1711770c1"
server_key_version: "encryptionkey@0"
I20260620 18:42:58.681295 8902 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:42:58.685057 8917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:58.686525 8902 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:42:58.686689 8902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980676253068-24012-0/minicluster-data/ts-1/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980676253068-24012-0/minicluster-data/ts-1/wal
uuid: "924265114bec4e2c9b822fdd707050b0"
format_stamp: "Formatted at 2026-06-20 18:42:58 on dist-test-slave-d645"
server_key: "10f5ca7f35f936e7b6fe13dfe5a33230"
server_key_iv: "ee3330e15e274c1e2a3a4ec1711770c1"
server_key_version: "encryptionkey@0"
I20260620 18:42:58.686877 8902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980676253068-24012-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980676253068-24012-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980676253068-24012-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:42:58.729019 8902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:42:58.733325 8902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:42:58.733628 8902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:42:58.734598 8902 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:42:58.736083 8902 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:42:58.736166 8902 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:58.736285 8902 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:42:58.736338 8902 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:58.768132 8902 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.2:40785
I20260620 18:42:58.768321 9035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.2:40785 every 8 connection(s)
I20260620 18:42:58.769407 8902 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980676253068-24012-0/minicluster-data/ts-1/data/info.pb
I20260620 18:42:58.773216 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 8902
I20260620 18:42:58.773414 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980676253068-24012-0/minicluster-data/ts-1/wal/instance
I20260620 18:42:58.773996 24012 external_mini_cluster.cc:1428] Setting key 3adfe0551fd31ccd9cd439f5cf89181a
Jun 20 18:42:58 dist-test-slave-d645 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980978, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.2@KRBTEST.COM for kudu/127.23.115.62@KRBTEST.COM
I20260620 18:42:58.790184 9038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:58.771998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.2:57953 (local address 127.23.115.62:40995)
0620 18:42:58.772455 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:58.772462 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:58.773550 (+ 1088us) server_negotiation.cc:407] Connection header received
0620 18:42:58.775813 (+ 2263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:58.775819 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:58.775902 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:58.775986 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:58.777629 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:58.778597 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:58.780255 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:58.780824 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:58.780960 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:58.784594 (+ 3634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:58.784599 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:58.784603 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:58.784637 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:58.787216 (+ 2579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:58.787940 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:58.787944 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:58.787947 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:58.788077 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:58.788483 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:58.788486 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:58.788488 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:58.788767 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:58.788900 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:58.789592 (+ 692us) server_negotiation.cc:299] Negotiation successful
0620 18:42:58.789816 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":205,"threads_started":1}
I20260620 18:42:58.791257 9039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:58.772516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:40995 (local address 127.23.115.2:57953)
0620 18:42:58.773387 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:58.773449 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:58.775534 (+ 2085us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:58.776222 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:58.776230 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:58.776850 (+ 620us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:42:58.777459 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:58.777476 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:58.778837 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:58.778841 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:58.780124 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:58.780137 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:58.781054 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:58.781059 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:58.781297 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:58.781950 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:42:58.781973 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:42:58.784357 (+ 2384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:42:58.787355 (+ 2998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:58.787365 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:58.787382 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:58.787813 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:58.788181 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:58.788186 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:58.788189 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:58.788369 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:58.788895 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:42:58.788903 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:42:58.789230 (+ 327us) client_negotiation.cc:769] Sending connection context
0620 18:42:58.789469 (+ 239us) client_negotiation.cc:240] Negotiation successful
0620 18:42:58.790039 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":335,"threads_started":1}
I20260620 18:42:58.792778 9036 heartbeater.cc:347] Connected to a master server at 127.23.115.62:40995
I20260620 18:42:58.793061 9036 heartbeater.cc:464] Registering TS with master...
I20260620 18:42:58.793855 9036 heartbeater.cc:511] Master 127.23.115.62:40995 requested a full tablet report, sending...
I20260620 18:42:58.795419 8707 ts_manager.cc:194] Registered new tserver with Master: 924265114bec4e2c9b822fdd707050b0 (127.23.115.2:40785)
I20260620 18:42:58.796582 8707 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.115.2@KRBTEST.COM'} at 127.23.115.2:57953
WARNING: no policy specified for kudu/127.23.115.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.115.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.115.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.3.keytab.
Entry for principal kudu/127.23.115.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.3.keytab.
I20260620 18:42:58.826390 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980676253068-24012-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980676253068-24012-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980676253068-24012-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980676253068-24012-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.115.3:0
--local_ip_for_outbound_sockets=127.23.115.3
--webserver_interface=127.23.115.3
--webserver_port=0
--tserver_master_addrs=127.23.115.62:40995
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.3.keytab
--principal=kudu/127.23.115.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:46561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:42:59.159526 9042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:42:59.159821 9042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:42:59.159930 9042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:42:59.168026 9042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:42:59.168190 9042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:42:59.168323 9042 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.115.3
I20260620 18:42:59.178462 9042 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:46561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980676253068-24012-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980676253068-24012-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.3.keytab
--principal=kudu/127.23.115.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.115.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980676253068-24012-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.115.3
--webserver_port=0
--tserver_master_addrs=127.23.115.62:40995
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.115.3
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980676253068-24012-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:42:59.179772 9042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:42:59.180987 9042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:42:59.193972 9047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:59.193997 9050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:42:59.194226 9048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:42:59.194639 9042 server_base.cc:1034] running on GCE node
I20260620 18:42:59.195305 9042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:42:59.196264 9042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:42:59.197480 9042 hybrid_clock.cc:648] HybridClock initialized: now 1781980979197434 us; error 37 us; skew 500 ppm
I20260620 18:42:59.197822 9042 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:42:59 dist-test-slave-d645 krb5kdc[8655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980979, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:42:59.200982 9042 init.cc:377] Logged in from keytab as kudu/127.23.115.3@KRBTEST.COM (short username kudu)
I20260620 18:42:59.202454 9042 webserver.cc:460] Webserver started at http://127.23.115.3:34971/ using document root <none> and password file <none>
I20260620 18:42:59.203164 9042 fs_manager.cc:359] Metadata directory not provided
I20260620 18:42:59.203289 9042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:42:59.203581 9042 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:42:59.206415 9042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980676253068-24012-0/minicluster-data/ts-2/data/instance:
uuid: "d0123d75f76848629c0c99149a1e285e"
format_stamp: "Formatted at 2026-06-20 18:42:59 on dist-test-slave-d645"
server_key: "afe1400359a8dfd477e75233a2496db7"
server_key_iv: "1ab35b33bb239895cb24d36210aae3f1"
server_key_version: "encryptionkey@0"
I20260620 18:42:59.207186 9042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980676253068-24012-0/minicluster-data/ts-2/wal/instance:
uuid: "d0123d75f76848629c0c99149a1e285e"
format_stamp: "Formatted at 2026-06-20 18:42:59 on dist-test-slave-d645"
server_key: "afe1400359a8dfd477e75233a2496db7"
server_key_iv: "1ab35b33bb239895cb24d36210aae3f1"
server_key_version: "encryptionkey@0"
I20260620 18:42:59.212857 9042 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260620 18:42:59.216545 9057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:59.217864 9042 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 18:42:59.218030 9042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980676253068-24012-0/minicluster-data/ts-2/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980676253068-24012-0/minicluster-data/ts-2/wal
uuid: "d0123d75f76848629c0c99149a1e285e"
format_stamp: "Formatted at 2026-06-20 18:42:59 on dist-test-slave-d645"
server_key: "afe1400359a8dfd477e75233a2496db7"
server_key_iv: "1ab35b33bb239895cb24d36210aae3f1"
server_key_version: "encryptionkey@0"
I20260620 18:42:59.218225 9042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980676253068-24012-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980676253068-24012-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980676253068-24012-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:42:59.231261 9042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:42:59.236182 9042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:42:59.236491 9042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:42:59.237661 9042 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:42:59.239308 9042 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:42:59.239393 9042 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:59.239508 9042 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:42:59.239559 9042 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:42:59.268412 8896 heartbeater.cc:503] Master 127.23.115.62:40995 was elected leader, sending a full tablet report...
I20260620 18:42:59.272799 9042 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.3:36811
I20260620 18:42:59.272874 9169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.3:36811 every 8 connection(s)
I20260620 18:42:59.274009 9042 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980676253068-24012-0/minicluster-data/ts-2/data/info.pb
I20260620 18:42:59.280351 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 9042
I20260620 18:42:59.280558 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980676253068-24012-0/minicluster-data/ts-2/wal/instance
I20260620 18:42:59.281049 24012 external_mini_cluster.cc:1428] Setting key 85cb6a297382f5fe5dcd78198863479d
Jun 20 18:42:59 dist-test-slave-d645 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980979, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.3@KRBTEST.COM for kudu/127.23.115.62@KRBTEST.COM
I20260620 18:42:59.297297 9038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:59.276877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.3:33329 (local address 127.23.115.62:40995)
0620 18:42:59.277035 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:59.277039 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:59.278576 (+ 1537us) server_negotiation.cc:407] Connection header received
0620 18:42:59.280440 (+ 1864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:59.280447 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:59.280540 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:59.280665 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:59.282251 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:59.283354 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:59.285560 (+ 2206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:59.286009 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:59.286125 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:59.290266 (+ 4141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:59.290272 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:59.290276 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:59.290315 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:59.293289 (+ 2974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:59.294297 (+ 1008us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:59.294304 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:59.294306 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:59.294544 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:59.295395 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:59.295401 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:59.295403 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:59.295764 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:59.295913 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:59.296703 (+ 790us) server_negotiation.cc:299] Negotiation successful
0620 18:42:59.296990 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 18:42:59.298291 9172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:59.277561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:40995 (local address 127.23.115.3:33329)
0620 18:42:59.278374 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:59.278417 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:59.280159 (+ 1742us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:59.280874 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:59.280883 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:59.281424 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:42:59.282043 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:59.282067 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:59.283523 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:59.283528 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:59.285406 (+ 1878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:59.285420 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:59.286246 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:59.286254 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:59.286470 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:59.287266 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:42:59.287308 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:42:59.290026 (+ 2718us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:42:59.293448 (+ 3422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:59.293459 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:59.293477 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:59.293945 (+ 468us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:59.294690 (+ 745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:42:59.294697 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:42:59.294701 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:42:59.295253 (+ 552us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:42:59.295907 (+ 654us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:42:59.295916 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:42:59.296299 (+ 383us) client_negotiation.cc:769] Sending connection context
0620 18:42:59.296541 (+ 242us) client_negotiation.cc:240] Negotiation successful
0620 18:42:59.297095 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":340,"threads_started":1}
I20260620 18:42:59.299945 9170 heartbeater.cc:347] Connected to a master server at 127.23.115.62:40995
I20260620 18:42:59.300252 9170 heartbeater.cc:464] Registering TS with master...
I20260620 18:42:59.300938 9170 heartbeater.cc:511] Master 127.23.115.62:40995 requested a full tablet report, sending...
I20260620 18:42:59.302380 8707 ts_manager.cc:194] Registered new tserver with Master: d0123d75f76848629c0c99149a1e285e (127.23.115.3:36811)
I20260620 18:42:59.303242 8707 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.115.3@KRBTEST.COM'} at 127.23.115.3:33329
I20260620 18:42:59.309386 24012 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:42:59.328264 9038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:59.314362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35398 (local address 127.23.115.62:40995)
0620 18:42:59.314607 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:59.314613 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:59.315042 (+ 429us) server_negotiation.cc:407] Connection header received
0620 18:42:59.315208 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:59.315213 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:59.315292 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:59.315378 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:59.316806 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:59.317541 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:59.318968 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:59.319373 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:59.319481 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:59.322176 (+ 2695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:59.322185 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:59.322189 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:59.322271 (+ 82us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:59.325495 (+ 3224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:59.326244 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:59.326248 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:59.326251 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:59.326383 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:59.326727 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:59.326731 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:59.326739 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:59.327112 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:59.327358 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:59.327716 (+ 358us) server_negotiation.cc:299] Negotiation successful
0620 18:42:59.327933 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260620 18:42:59.332425 8707 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35398:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:42:59.339169 8707 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:42:59.365545 9184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:59.354124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36282 (local address 127.23.115.3:36811)
0620 18:42:59.354870 (+ 746us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:59.354879 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:59.355255 (+ 376us) server_negotiation.cc:407] Connection header received
0620 18:42:59.355599 (+ 344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:59.355607 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:59.355820 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:59.356006 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:42:59.357231 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:59.359209 (+ 1978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:59.362009 (+ 2800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:59.363812 (+ 1803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:59.363925 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:59.364209 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:59.364938 (+ 729us) server_negotiation.cc:299] Negotiation successful
0620 18:42:59.365129 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":402,"threads_started":1}
I20260620 18:42:59.365767 9182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:59.353169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.1:43789 (local address 127.0.0.1:48016)
0620 18:42:59.353951 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:59.353994 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:59.354185 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:59.356078 (+ 1893us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:59.356085 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:59.356114 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:42:59.356580 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:59.356593 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:59.358836 (+ 2243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:59.358841 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:59.362798 (+ 3957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:59.362812 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:59.364820 (+ 2008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:59.364827 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:59.365030 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:59.365067 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 18:42:59.365131 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 18:42:59.365343 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":316,"threads_started":1}
I20260620 18:42:59.366735 9185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:59.353638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.3:36811 (local address 127.0.0.1:36282)
0620 18:42:59.355151 (+ 1513us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:59.355171 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:59.355369 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:59.356379 (+ 1010us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:59.356385 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:59.356421 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:42:59.356984 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:59.356997 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:59.359421 (+ 2424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:59.359427 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:59.361802 (+ 2375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:59.361820 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:59.364410 (+ 2590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:59.364417 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:59.364641 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:59.364691 (+ 50us) client_negotiation.cc:769] Sending connection context
0620 18:42:59.366186 (+ 1495us) client_negotiation.cc:240] Negotiation successful
0620 18:42:59.366335 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1376,"mutex_wait_us":729,"thread_start_us":452,"threads_started":1}
I20260620 18:42:59.367311 9183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:59.353328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48016 (local address 127.23.115.1:43789)
0620 18:42:59.355346 (+ 2018us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:59.355354 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:59.355379 (+ 25us) server_negotiation.cc:407] Connection header received
0620 18:42:59.355475 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:59.355481 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:59.355755 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:59.356803 (+ 1048us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:42:59.357018 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:59.358696 (+ 1678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:59.363037 (+ 4341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:59.364548 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:59.366586 (+ 2038us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:59.366761 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:59.366825 (+ 64us) server_negotiation.cc:299] Negotiation successful
0620 18:42:59.366999 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1855,"thread_start_us":2655,"threads_started":1}
I20260620 18:42:59.370586 9105 tablet_service.cc:1467] Processing CreateTablet for tablet ccedf58aa71e4990a179e613a2812cae (DEFAULT_TABLE table=test-table [id=6b947bd3494e46b3a0a3245aab5c23de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:42:59.370599 8831 tablet_service.cc:1467] Processing CreateTablet for tablet ccedf58aa71e4990a179e613a2812cae (DEFAULT_TABLE table=test-table [id=6b947bd3494e46b3a0a3245aab5c23de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:42:59.371841 9105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccedf58aa71e4990a179e613a2812cae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:42:59.371912 8831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccedf58aa71e4990a179e613a2812cae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:42:59.374605 9182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:59.355271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.2:40785 (local address 127.0.0.1:41508)
0620 18:42:59.366022 (+ 10751us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:59.366042 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:59.366153 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:59.367894 (+ 1741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:59.367899 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:59.367926 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:42:59.368078 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:59.368089 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:59.370067 (+ 1978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:59.370072 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:59.371808 (+ 1736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:59.371823 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:59.373832 (+ 2009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:59.373838 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:59.374028 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:59.374068 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 18:42:59.374170 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 18:42:59.374307 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10644,"thread_start_us":11073,"threads_started":1}
I20260620 18:42:59.375423 9187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:59.355723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41508 (local address 127.23.115.2:40785)
0620 18:42:59.367289 (+ 11566us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:59.367297 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:59.367319 (+ 22us) server_negotiation.cc:407] Connection header received
0620 18:42:59.367402 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:59.367411 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:59.367678 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:59.368260 (+ 582us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:42:59.368473 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:59.369911 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:59.372021 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:59.373388 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:59.373527 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:59.373714 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:59.374901 (+ 1187us) server_negotiation.cc:299] Negotiation successful
0620 18:42:59.375108 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11411,"thread_start_us":11302,"threads_started":1}
I20260620 18:42:59.379136 8971 tablet_service.cc:1467] Processing CreateTablet for tablet ccedf58aa71e4990a179e613a2812cae (DEFAULT_TABLE table=test-table [id=6b947bd3494e46b3a0a3245aab5c23de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:42:59.380520 8971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccedf58aa71e4990a179e613a2812cae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:42:59.383865 9188 tablet_bootstrap.cc:492] T ccedf58aa71e4990a179e613a2812cae P c848eed8b6d44403a46a73dbc59cb289: Bootstrap starting.
I20260620 18:42:59.385845 9189 tablet_bootstrap.cc:492] T ccedf58aa71e4990a179e613a2812cae P d0123d75f76848629c0c99149a1e285e: Bootstrap starting.
I20260620 18:42:59.387496 9188 tablet_bootstrap.cc:654] T ccedf58aa71e4990a179e613a2812cae P c848eed8b6d44403a46a73dbc59cb289: Neither blocks nor log segments found. Creating new log.
I20260620 18:42:59.388721 9188 log.cc:824] T ccedf58aa71e4990a179e613a2812cae P c848eed8b6d44403a46a73dbc59cb289: Log is configured to *not* fsync() on all Append() calls
I20260620 18:42:59.389771 9189 tablet_bootstrap.cc:654] T ccedf58aa71e4990a179e613a2812cae P d0123d75f76848629c0c99149a1e285e: Neither blocks nor log segments found. Creating new log.
I20260620 18:42:59.391016 9189 log.cc:824] T ccedf58aa71e4990a179e613a2812cae P d0123d75f76848629c0c99149a1e285e: Log is configured to *not* fsync() on all Append() calls
I20260620 18:42:59.392130 9188 tablet_bootstrap.cc:492] T ccedf58aa71e4990a179e613a2812cae P c848eed8b6d44403a46a73dbc59cb289: No bootstrap required, opened a new log
I20260620 18:42:59.392444 9188 ts_tablet_manager.cc:1397] T ccedf58aa71e4990a179e613a2812cae P c848eed8b6d44403a46a73dbc59cb289: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260620 18:42:59.393101 9192 tablet_bootstrap.cc:492] T ccedf58aa71e4990a179e613a2812cae P 924265114bec4e2c9b822fdd707050b0: Bootstrap starting.
I20260620 18:42:59.393738 9189 tablet_bootstrap.cc:492] T ccedf58aa71e4990a179e613a2812cae P d0123d75f76848629c0c99149a1e285e: No bootstrap required, opened a new log
I20260620 18:42:59.394021 9189 ts_tablet_manager.cc:1397] T ccedf58aa71e4990a179e613a2812cae P d0123d75f76848629c0c99149a1e285e: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260620 18:42:59.397428 9192 tablet_bootstrap.cc:654] T ccedf58aa71e4990a179e613a2812cae P 924265114bec4e2c9b822fdd707050b0: Neither blocks nor log segments found. Creating new log.
I20260620 18:42:59.398653 9192 log.cc:824] T ccedf58aa71e4990a179e613a2812cae P 924265114bec4e2c9b822fdd707050b0: Log is configured to *not* fsync() on all Append() calls
I20260620 18:42:59.401077 9188 raft_consensus.cc:348] T ccedf58aa71e4990a179e613a2812cae P c848eed8b6d44403a46a73dbc59cb289 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c848eed8b6d44403a46a73dbc59cb289" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 43789 } } peers { permanent_uuid: "d0123d75f76848629c0c99149a1e285e" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 36811 } } peers { permanent_uuid: "924265114bec4e2c9b822fdd707050b0" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 40785 } }
I20260620 18:42:59.401660 9188 raft_consensus.cc:374] T ccedf58aa71e4990a179e613a2812cae P c848eed8b6d44403a46a73dbc59cb289 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:42:59.401779 9188 raft_consensus.cc:729] T ccedf58aa71e4990a179e613a2812cae P c848eed8b6d44403a46a73dbc59cb289 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c848eed8b6d44403a46a73dbc59cb289, State: Initialized, Role: FOLLOWER
I20260620 18:42:59.401866 9192 tablet_bootstrap.cc:492] T ccedf58aa71e4990a179e613a2812cae P 924265114bec4e2c9b822fdd707050b0: No bootstrap required, opened a new log
I20260620 18:42:59.402159 9192 ts_tablet_manager.cc:1397] T ccedf58aa71e4990a179e613a2812cae P 924265114bec4e2c9b822fdd707050b0: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20260620 18:42:59.402433 9188 consensus_queue.cc:260] T ccedf58aa71e4990a179e613a2812cae P c848eed8b6d44403a46a73dbc59cb289 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c848eed8b6d44403a46a73dbc59cb289" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 43789 } } peers { permanent_uuid: "d0123d75f76848629c0c99149a1e285e" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 36811 } } peers { permanent_uuid: "924265114bec4e2c9b822fdd707050b0" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 40785 } }
I20260620 18:42:59.404201 9188 ts_tablet_manager.cc:1428] T ccedf58aa71e4990a179e613a2812cae P c848eed8b6d44403a46a73dbc59cb289: Time spent starting tablet: real 0.012s user 0.009s sys 0.002s
I20260620 18:42:59.404047 9189 raft_consensus.cc:348] T ccedf58aa71e4990a179e613a2812cae P d0123d75f76848629c0c99149a1e285e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c848eed8b6d44403a46a73dbc59cb289" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 43789 } } peers { permanent_uuid: "d0123d75f76848629c0c99149a1e285e" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 36811 } } peers { permanent_uuid: "924265114bec4e2c9b822fdd707050b0" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 40785 } }
I20260620 18:42:59.404544 9189 raft_consensus.cc:374] T ccedf58aa71e4990a179e613a2812cae P d0123d75f76848629c0c99149a1e285e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:42:59.404654 9189 raft_consensus.cc:729] T ccedf58aa71e4990a179e613a2812cae P d0123d75f76848629c0c99149a1e285e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0123d75f76848629c0c99149a1e285e, State: Initialized, Role: FOLLOWER
I20260620 18:42:59.405223 9189 consensus_queue.cc:260] T ccedf58aa71e4990a179e613a2812cae P d0123d75f76848629c0c99149a1e285e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c848eed8b6d44403a46a73dbc59cb289" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 43789 } } peers { permanent_uuid: "d0123d75f76848629c0c99149a1e285e" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 36811 } } peers { permanent_uuid: "924265114bec4e2c9b822fdd707050b0" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 40785 } }
I20260620 18:42:59.406199 9170 heartbeater.cc:503] Master 127.23.115.62:40995 was elected leader, sending a full tablet report...
I20260620 18:42:59.406822 9189 ts_tablet_manager.cc:1428] T ccedf58aa71e4990a179e613a2812cae P d0123d75f76848629c0c99149a1e285e: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260620 18:42:59.411203 9192 raft_consensus.cc:348] T ccedf58aa71e4990a179e613a2812cae P 924265114bec4e2c9b822fdd707050b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c848eed8b6d44403a46a73dbc59cb289" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 43789 } } peers { permanent_uuid: "d0123d75f76848629c0c99149a1e285e" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 36811 } } peers { permanent_uuid: "924265114bec4e2c9b822fdd707050b0" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 40785 } }
I20260620 18:42:59.411562 9192 raft_consensus.cc:374] T ccedf58aa71e4990a179e613a2812cae P 924265114bec4e2c9b822fdd707050b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:42:59.411646 9192 raft_consensus.cc:729] T ccedf58aa71e4990a179e613a2812cae P 924265114bec4e2c9b822fdd707050b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 924265114bec4e2c9b822fdd707050b0, State: Initialized, Role: FOLLOWER
I20260620 18:42:59.412088 9192 consensus_queue.cc:260] T ccedf58aa71e4990a179e613a2812cae P 924265114bec4e2c9b822fdd707050b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c848eed8b6d44403a46a73dbc59cb289" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 43789 } } peers { permanent_uuid: "d0123d75f76848629c0c99149a1e285e" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 36811 } } peers { permanent_uuid: "924265114bec4e2c9b822fdd707050b0" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 40785 } }
I20260620 18:42:59.412992 9036 heartbeater.cc:503] Master 127.23.115.62:40995 was elected leader, sending a full tablet report...
I20260620 18:42:59.413587 9192 ts_tablet_manager.cc:1428] T ccedf58aa71e4990a179e613a2812cae P 924265114bec4e2c9b822fdd707050b0: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
W20260620 18:42:59.487175 8897 tablet.cc:2369] T ccedf58aa71e4990a179e613a2812cae P c848eed8b6d44403a46a73dbc59cb289: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:42:59.521618 9037 tablet.cc:2369] T ccedf58aa71e4990a179e613a2812cae P 924265114bec4e2c9b822fdd707050b0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:42:59.525730 9171 tablet.cc:2369] T ccedf58aa71e4990a179e613a2812cae P d0123d75f76848629c0c99149a1e285e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:42:59.725317 9196 raft_consensus.cc:482] T ccedf58aa71e4990a179e613a2812cae P 924265114bec4e2c9b822fdd707050b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:42:59.725598 9196 raft_consensus.cc:504] T ccedf58aa71e4990a179e613a2812cae P 924265114bec4e2c9b822fdd707050b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c848eed8b6d44403a46a73dbc59cb289" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 43789 } } peers { permanent_uuid: "d0123d75f76848629c0c99149a1e285e" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 36811 } } peers { permanent_uuid: "924265114bec4e2c9b822fdd707050b0" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 40785 } }
I20260620 18:42:59.727336 9196 leader_election.cc:288] T ccedf58aa71e4990a179e613a2812cae P 924265114bec4e2c9b822fdd707050b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c848eed8b6d44403a46a73dbc59cb289 (127.23.115.1:43789), d0123d75f76848629c0c99149a1e285e (127.23.115.3:36811)
I20260620 18:42:59.735896 9184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:59.728170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.2:50123 (local address 127.23.115.3:36811)
0620 18:42:59.728445 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:59.728452 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:59.729103 (+ 651us) server_negotiation.cc:407] Connection header received
0620 18:42:59.729316 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:59.729321 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:59.729391 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:59.729466 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:42:59.730279 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:59.731229 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:59.733422 (+ 2193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:59.734586 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:59.734722 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:59.734786 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:59.735569 (+ 783us) server_negotiation.cc:299] Negotiation successful
0620 18:42:59.735686 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
I20260620 18:42:59.736081 9197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:59.727878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.3:36811 (local address 127.23.115.2:50123)
0620 18:42:59.728959 (+ 1081us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:59.728977 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:59.729169 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:59.729525 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:59.729530 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:59.729557 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:42:59.730069 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:59.730082 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:59.731352 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:59.731358 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:59.733206 (+ 1848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:59.733219 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:59.734886 (+ 1667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:59.734895 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:59.735141 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:59.735191 (+ 50us) client_negotiation.cc:769] Sending connection context
0620 18:42:59.735541 (+ 350us) client_negotiation.cc:240] Negotiation successful
0620 18:42:59.735702 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":948,"thread_start_us":628,"threads_started":1}
I20260620 18:42:59.737048 9183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:59.727943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.2:33577 (local address 127.23.115.1:43789)
0620 18:42:59.728304 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:59.728313 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:59.729933 (+ 1620us) server_negotiation.cc:407] Connection header received
0620 18:42:59.730050 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:59.730055 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:59.730131 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:59.730219 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:42:59.730868 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:59.731907 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:59.734530 (+ 2623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:59.736171 (+ 1641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:59.736346 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:59.736421 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:59.736756 (+ 335us) server_negotiation.cc:299] Negotiation successful
0620 18:42:59.736873 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 18:42:59.737207 9198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:59.728353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.1:43789 (local address 127.23.115.2:33577)
0620 18:42:59.729802 (+ 1449us) negotiation.cc:107] Waiting for socket to connect
0620 18:42:59.729820 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:42:59.729934 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:42:59.730278 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:42:59.730283 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:42:59.730311 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:42:59.730669 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:59.730681 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:59.732163 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:59.732170 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:59.734220 (+ 2050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:42:59.734236 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:42:59.736455 (+ 2219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:42:59.736462 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:42:59.736592 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:59.736623 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 18:42:59.736713 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 18:42:59.736852 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1222,"mutex_wait_us":1,"thread_start_us":684,"threads_started":1}
I20260620 18:42:59.737196 9125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccedf58aa71e4990a179e613a2812cae" candidate_uuid: "924265114bec4e2c9b822fdd707050b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0123d75f76848629c0c99149a1e285e" is_pre_election: true
I20260620 18:42:59.737756 9125 raft_consensus.cc:2454] T ccedf58aa71e4990a179e613a2812cae P d0123d75f76848629c0c99149a1e285e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 924265114bec4e2c9b822fdd707050b0 in term 0.
I20260620 18:42:59.738206 8851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccedf58aa71e4990a179e613a2812cae" candidate_uuid: "924265114bec4e2c9b822fdd707050b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c848eed8b6d44403a46a73dbc59cb289" is_pre_election: true
I20260620 18:42:59.738404 8918 leader_election.cc:302] T ccedf58aa71e4990a179e613a2812cae P 924265114bec4e2c9b822fdd707050b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 924265114bec4e2c9b822fdd707050b0, d0123d75f76848629c0c99149a1e285e; no voters:
I20260620 18:42:59.738674 8851 raft_consensus.cc:2454] T ccedf58aa71e4990a179e613a2812cae P c848eed8b6d44403a46a73dbc59cb289 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 924265114bec4e2c9b822fdd707050b0 in term 0.
I20260620 18:42:59.738778 9196 raft_consensus.cc:2781] T ccedf58aa71e4990a179e613a2812cae P 924265114bec4e2c9b822fdd707050b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:42:59.738902 9196 raft_consensus.cc:482] T ccedf58aa71e4990a179e613a2812cae P 924265114bec4e2c9b822fdd707050b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:42:59.739035 9196 raft_consensus.cc:3037] T ccedf58aa71e4990a179e613a2812cae P 924265114bec4e2c9b822fdd707050b0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:42:59.741294 9196 raft_consensus.cc:504] T ccedf58aa71e4990a179e613a2812cae P 924265114bec4e2c9b822fdd707050b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c848eed8b6d44403a46a73dbc59cb289" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 43789 } } peers { permanent_uuid: "d0123d75f76848629c0c99149a1e285e" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 36811 } } peers { permanent_uuid: "924265114bec4e2c9b822fdd707050b0" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 40785 } }
I20260620 18:42:59.741870 9196 leader_election.cc:288] T ccedf58aa71e4990a179e613a2812cae P 924265114bec4e2c9b822fdd707050b0 [CANDIDATE]: Term 1 election: Requested vote from peers c848eed8b6d44403a46a73dbc59cb289 (127.23.115.1:43789), d0123d75f76848629c0c99149a1e285e (127.23.115.3:36811)
I20260620 18:42:59.742452 9125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccedf58aa71e4990a179e613a2812cae" candidate_uuid: "924265114bec4e2c9b822fdd707050b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0123d75f76848629c0c99149a1e285e"
I20260620 18:42:59.742530 8851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccedf58aa71e4990a179e613a2812cae" candidate_uuid: "924265114bec4e2c9b822fdd707050b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c848eed8b6d44403a46a73dbc59cb289"
I20260620 18:42:59.742638 9125 raft_consensus.cc:3037] T ccedf58aa71e4990a179e613a2812cae P d0123d75f76848629c0c99149a1e285e [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:42:59.742744 8851 raft_consensus.cc:3037] T ccedf58aa71e4990a179e613a2812cae P c848eed8b6d44403a46a73dbc59cb289 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:42:59.745302 9125 raft_consensus.cc:2454] T ccedf58aa71e4990a179e613a2812cae P d0123d75f76848629c0c99149a1e285e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 924265114bec4e2c9b822fdd707050b0 in term 1.
I20260620 18:42:59.745999 8918 leader_election.cc:302] T ccedf58aa71e4990a179e613a2812cae P 924265114bec4e2c9b822fdd707050b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 924265114bec4e2c9b822fdd707050b0, d0123d75f76848629c0c99149a1e285e; no voters:
I20260620 18:42:59.745965 8851 raft_consensus.cc:2454] T ccedf58aa71e4990a179e613a2812cae P c848eed8b6d44403a46a73dbc59cb289 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 924265114bec4e2c9b822fdd707050b0 in term 1.
I20260620 18:42:59.746404 9196 raft_consensus.cc:2781] T ccedf58aa71e4990a179e613a2812cae P 924265114bec4e2c9b822fdd707050b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:42:59.747014 9196 raft_consensus.cc:686] T ccedf58aa71e4990a179e613a2812cae P 924265114bec4e2c9b822fdd707050b0 [term 1 LEADER]: Becoming Leader. State: Replica: 924265114bec4e2c9b822fdd707050b0, State: Running, Role: LEADER
I20260620 18:42:59.747552 9196 consensus_queue.cc:237] T ccedf58aa71e4990a179e613a2812cae P 924265114bec4e2c9b822fdd707050b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c848eed8b6d44403a46a73dbc59cb289" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 43789 } } peers { permanent_uuid: "d0123d75f76848629c0c99149a1e285e" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 36811 } } peers { permanent_uuid: "924265114bec4e2c9b822fdd707050b0" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 40785 } }
I20260620 18:42:59.752158 8707 catalog_manager.cc:5495] T ccedf58aa71e4990a179e613a2812cae P 924265114bec4e2c9b822fdd707050b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 924265114bec4e2c9b822fdd707050b0 (127.23.115.2). New cstate: current_term: 1 leader_uuid: "924265114bec4e2c9b822fdd707050b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c848eed8b6d44403a46a73dbc59cb289" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 43789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0123d75f76848629c0c99149a1e285e" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 36811 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "924265114bec4e2c9b822fdd707050b0" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 40785 } health_report { overall_health: HEALTHY } } }
I20260620 18:42:59.818078 9187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:59.813044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41512 (local address 127.23.115.2:40785)
0620 18:42:59.813255 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:59.813261 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:59.813278 (+ 17us) server_negotiation.cc:407] Connection header received
0620 18:42:59.813369 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:59.813373 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:59.813452 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:59.813577 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:42:59.813943 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:59.814623 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:59.816044 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:59.816434 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:59.816539 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:59.817026 (+ 487us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:42:59.817510 (+ 484us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:42:59.817635 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:59.817737 (+ 102us) server_negotiation.cc:299] Negotiation successful
0620 18:42:59.817816 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 18:42:59.826896 9125 raft_consensus.cc:1261] T ccedf58aa71e4990a179e613a2812cae P d0123d75f76848629c0c99149a1e285e [term 1 FOLLOWER]: Refusing update from remote peer 924265114bec4e2c9b822fdd707050b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:42:59.826922 8851 raft_consensus.cc:1261] T ccedf58aa71e4990a179e613a2812cae P c848eed8b6d44403a46a73dbc59cb289 [term 1 FOLLOWER]: Refusing update from remote peer 924265114bec4e2c9b822fdd707050b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:42:59.827841 9199 consensus_queue.cc:1035] T ccedf58aa71e4990a179e613a2812cae P 924265114bec4e2c9b822fdd707050b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0123d75f76848629c0c99149a1e285e" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 36811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:42:59.828197 9196 consensus_queue.cc:1035] T ccedf58aa71e4990a179e613a2812cae P 924265114bec4e2c9b822fdd707050b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c848eed8b6d44403a46a73dbc59cb289" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 43789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:42:59.839790 9204 mvcc.cc:205] Tried to move back new op lower bound from 7298994093361590272 to 7298994093050368000. Current Snapshot: MvccSnapshot[applied={T|T < 7298994093361590272}]
I20260620 18:42:59.851393 8707 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35398:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:42:59.851668 8707 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35398:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:42:59.854907 8707 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f1002ae35eca41f8ae3127886d3c5b6b: Sending DeleteTablet for 3 replicas of tablet ccedf58aa71e4990a179e613a2812cae
I20260620 18:42:59.855985 8831 tablet_service.cc:1514] Processing DeleteTablet for tablet ccedf58aa71e4990a179e613a2812cae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:42:59 UTC) from {username='kudu', principal='kudu/127.23.115.62@KRBTEST.COM'} at 127.0.0.1:48016
I20260620 18:42:59.856074 9105 tablet_service.cc:1514] Processing DeleteTablet for tablet ccedf58aa71e4990a179e613a2812cae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:42:59 UTC) from {username='kudu', principal='kudu/127.23.115.62@KRBTEST.COM'} at 127.0.0.1:36282
I20260620 18:42:59.856074 8971 tablet_service.cc:1514] Processing DeleteTablet for tablet ccedf58aa71e4990a179e613a2812cae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:42:59 UTC) from {username='kudu', principal='kudu/127.23.115.62@KRBTEST.COM'} at 127.0.0.1:41508
I20260620 18:42:59.862535 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 8768
I20260620 18:42:59.866901 9211 tablet_replica.cc:330] T ccedf58aa71e4990a179e613a2812cae P c848eed8b6d44403a46a73dbc59cb289: stopping tablet replica
I20260620 18:42:59.868193 9211 raft_consensus.cc:2229] T ccedf58aa71e4990a179e613a2812cae P c848eed8b6d44403a46a73dbc59cb289 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:42:59.868752 9211 raft_consensus.cc:2258] T ccedf58aa71e4990a179e613a2812cae P c848eed8b6d44403a46a73dbc59cb289 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:42:59.870556 9213 tablet_replica.cc:330] T ccedf58aa71e4990a179e613a2812cae P 924265114bec4e2c9b822fdd707050b0: stopping tablet replica
I20260620 18:42:59.871006 9213 raft_consensus.cc:2229] T ccedf58aa71e4990a179e613a2812cae P 924265114bec4e2c9b822fdd707050b0 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:42:59.871722 9213 raft_consensus.cc:2258] T ccedf58aa71e4990a179e613a2812cae P 924265114bec4e2c9b822fdd707050b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:42:59.874145 9211 ts_tablet_manager.cc:1905] T ccedf58aa71e4990a179e613a2812cae P c848eed8b6d44403a46a73dbc59cb289: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:42:59.879287 9213 ts_tablet_manager.cc:1905] T ccedf58aa71e4990a179e613a2812cae P 924265114bec4e2c9b822fdd707050b0: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 20 18:42:59 dist-test-slave-d645 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980977, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.115.1@KRBTEST.COM
I20260620 18:42:59.890115 9211 ts_tablet_manager.cc:1918] T ccedf58aa71e4990a179e613a2812cae P c848eed8b6d44403a46a73dbc59cb289: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:42:59.890297 9211 log.cc:1196] T ccedf58aa71e4990a179e613a2812cae P c848eed8b6d44403a46a73dbc59cb289: Deleting WAL directory at /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980676253068-24012-0/minicluster-data/ts-0/wal/wals/ccedf58aa71e4990a179e613a2812cae
I20260620 18:42:59.890852 9211 ts_tablet_manager.cc:1939] T ccedf58aa71e4990a179e613a2812cae P c848eed8b6d44403a46a73dbc59cb289: Deleting consensus metadata
I20260620 18:42:59.891458 9212 tablet_replica.cc:330] T ccedf58aa71e4990a179e613a2812cae P d0123d75f76848629c0c99149a1e285e: stopping tablet replica
I20260620 18:42:59.891904 9212 raft_consensus.cc:2229] T ccedf58aa71e4990a179e613a2812cae P d0123d75f76848629c0c99149a1e285e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:42:59.892474 9212 raft_consensus.cc:2258] T ccedf58aa71e4990a179e613a2812cae P d0123d75f76848629c0c99149a1e285e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:42:59.894471 9212 ts_tablet_manager.cc:1905] T ccedf58aa71e4990a179e613a2812cae P d0123d75f76848629c0c99149a1e285e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:42:59.895313 9213 ts_tablet_manager.cc:1918] T ccedf58aa71e4990a179e613a2812cae P 924265114bec4e2c9b822fdd707050b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:42:59.895483 9213 log.cc:1196] T ccedf58aa71e4990a179e613a2812cae P 924265114bec4e2c9b822fdd707050b0: Deleting WAL directory at /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980676253068-24012-0/minicluster-data/ts-1/wal/wals/ccedf58aa71e4990a179e613a2812cae
I20260620 18:42:59.895951 9213 ts_tablet_manager.cc:1939] T ccedf58aa71e4990a179e613a2812cae P 924265114bec4e2c9b822fdd707050b0: Deleting consensus metadata
I20260620 18:42:59.897744 8691 catalog_manager.cc:4841] TS 924265114bec4e2c9b822fdd707050b0 (127.23.115.2:40785): tablet ccedf58aa71e4990a179e613a2812cae (table test-table [id=6b947bd3494e46b3a0a3245aab5c23de]) successfully deleted
I20260620 18:42:59.898491 9183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:42:59.865632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48026 (local address 127.23.115.1:43789)
0620 18:42:59.865835 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 18:42:59.865841 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:42:59.876147 (+ 10306us) server_negotiation.cc:407] Connection header received
0620 18:42:59.876670 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:42:59.876676 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:42:59.876762 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:42:59.876839 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:42:59.879011 (+ 2172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:59.880384 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:59.882876 (+ 2492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:42:59.883604 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:42:59.883714 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:42:59.889463 (+ 5749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:42:59.889479 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:42:59.889499 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:42:59.889551 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:42:59.893785 (+ 4234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:59.894885 (+ 1100us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:59.894894 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:59.894898 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:59.895092 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:42:59.895691 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:42:59.895697 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:42:59.895700 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:42:59.896206 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:42:59.896482 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0620 18:42:59.897131 (+ 649us) server_negotiation.cc:299] Negotiation successful
0620 18:42:59.897371 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 18:42:59.899277 8694 catalog_manager.cc:4841] TS c848eed8b6d44403a46a73dbc59cb289 (127.23.115.1:43789): tablet ccedf58aa71e4990a179e613a2812cae (table test-table [id=6b947bd3494e46b3a0a3245aab5c23de]) successfully deleted
I20260620 18:42:59.900648 8891 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:42:59.900840 9212 ts_tablet_manager.cc:1918] T ccedf58aa71e4990a179e613a2812cae P d0123d75f76848629c0c99149a1e285e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:42:59.900996 9212 log.cc:1196] T ccedf58aa71e4990a179e613a2812cae P d0123d75f76848629c0c99149a1e285e: Deleting WAL directory at /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980676253068-24012-0/minicluster-data/ts-2/wal/wals/ccedf58aa71e4990a179e613a2812cae
I20260620 18:42:59.901535 9212 ts_tablet_manager.cc:1939] T ccedf58aa71e4990a179e613a2812cae P d0123d75f76848629c0c99149a1e285e: Deleting consensus metadata
I20260620 18:42:59.903617 8691 catalog_manager.cc:4841] TS d0123d75f76848629c0c99149a1e285e (127.23.115.3:36811): tablet ccedf58aa71e4990a179e613a2812cae (table test-table [id=6b947bd3494e46b3a0a3245aab5c23de]) successfully deleted
I20260620 18:42:59.992450 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 8768
I20260620 18:43:00.012020 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 8902
Jun 20 18:43:00 dist-test-slave-d645 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980977, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.115.2@KRBTEST.COM
I20260620 18:43:00.031172 9187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:00.012965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41524 (local address 127.23.115.2:40785)
0620 18:43:00.013168 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:00.013173 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:00.013223 (+ 50us) server_negotiation.cc:407] Connection header received
0620 18:43:00.013519 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:00.013525 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:00.013622 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:00.013755 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:00.015475 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:00.016358 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:00.018908 (+ 2550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:00.019450 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:00.019559 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:00.024887 (+ 5328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:00.024901 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:00.024918 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:00.024968 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:00.028253 (+ 3285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:00.028965 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:00.028973 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:00.028975 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:00.029124 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:00.029506 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:00.029511 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:00.029513 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:00.029854 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:00.030208 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:00.030520 (+ 312us) server_negotiation.cc:299] Negotiation successful
0620 18:43:00.030749 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 18:43:00.032126 9031 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:43:00.149714 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 8902
I20260620 18:43:00.166085 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 9042
Jun 20 18:43:00 dist-test-slave-d645 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980977, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.115.3@KRBTEST.COM
I20260620 18:43:00.183611 9184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:00.167278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36296 (local address 127.23.115.3:36811)
0620 18:43:00.167488 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:00.167494 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:00.167551 (+ 57us) server_negotiation.cc:407] Connection header received
0620 18:43:00.167763 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:00.167768 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:00.167839 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:00.167943 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:00.169213 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:00.169956 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:00.171392 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:00.171827 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:00.171930 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:00.177263 (+ 5333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:00.177279 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:00.177295 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:00.177336 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:00.180922 (+ 3586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:00.181614 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:00.181621 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:00.181623 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:00.181788 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:00.182092 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:00.182095 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:00.182097 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:00.182440 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:00.182649 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:00.183078 (+ 429us) server_negotiation.cc:299] Negotiation successful
0620 18:43:00.183254 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 18:43:00.184217 9165 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:43:00.272238 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 9042
I20260620 18:43:00.287854 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 8668
I20260620 18:43:00.289115 8737 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:43:00.398036 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 8668
2026-06-20T18:43:00Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4512 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-20T18:43:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:43:00Z Disabled control of system clock
I20260620 18:43:00.470759 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980676253068-24012-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980676253068-24012-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980676253068-24012-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.115.62:40637
--webserver_interface=127.23.115.62
--webserver_port=0
--builtin_ntp_servers=127.23.115.20:35325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.115.62:40637
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 18:43:00.795394 9222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:43:00.795706 9222 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:43:00.795804 9222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:43:00.803295 9222 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:43:00.803423 9222 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:43:00.803478 9222 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:43:00.803565 9222 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:43:00.803617 9222 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:43:00.813457 9222 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:35325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980676253068-24012-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980676253068-24012-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.115.62:40637
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.115.62:40637
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.115.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980676253068-24012-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:43:00.814775 9222 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:43:00.816548 9222 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:43:00.828052 9227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:00.828258 9228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:00.828688 9230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:00.829252 9222 server_base.cc:1034] running on GCE node
I20260620 18:43:00.830035 9222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:43:00.831487 9222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:43:00.832705 9222 hybrid_clock.cc:648] HybridClock initialized: now 1781980980832642 us; error 41 us; skew 500 ppm
I20260620 18:43:00.833035 9222 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:43:00.838758 9222 webserver.cc:460] Webserver started at http://127.23.115.62:38769/ using document root <none> and password file <none>
I20260620 18:43:00.839534 9222 fs_manager.cc:359] Metadata directory not provided
I20260620 18:43:00.839641 9222 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:43:00.839932 9222 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:43:00.842792 9222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980676253068-24012-0/minicluster-data/master-0/data/instance:
uuid: "e1e74114c0814dac9c07d59a4ecf8657"
format_stamp: "Formatted at 2026-06-20 18:43:00 on dist-test-slave-d645"
server_key: "9ced6241a419544b89e7f1d7abcc17f8"
server_key_iv: "5f7fbee107f9ec2c65df1ebe76a42edd"
server_key_version: "encryptionkey@0"
I20260620 18:43:00.843621 9222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980676253068-24012-0/minicluster-data/master-0/wal/instance:
uuid: "e1e74114c0814dac9c07d59a4ecf8657"
format_stamp: "Formatted at 2026-06-20 18:43:00 on dist-test-slave-d645"
server_key: "9ced6241a419544b89e7f1d7abcc17f8"
server_key_iv: "5f7fbee107f9ec2c65df1ebe76a42edd"
server_key_version: "encryptionkey@0"
I20260620 18:43:00.848814 9222 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:43:00.852536 9236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:00.853945 9222 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 18:43:00.854106 9222 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980676253068-24012-0/minicluster-data/master-0/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980676253068-24012-0/minicluster-data/master-0/wal
uuid: "e1e74114c0814dac9c07d59a4ecf8657"
format_stamp: "Formatted at 2026-06-20 18:43:00 on dist-test-slave-d645"
server_key: "9ced6241a419544b89e7f1d7abcc17f8"
server_key_iv: "5f7fbee107f9ec2c65df1ebe76a42edd"
server_key_version: "encryptionkey@0"
I20260620 18:43:00.854314 9222 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980676253068-24012-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980676253068-24012-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980676253068-24012-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:43:00.882422 9222 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:43:00.883481 9222 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:43:00.883790 9222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:43:00.902714 9222 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.62:40637
I20260620 18:43:00.902738 9287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.62:40637 every 8 connection(s)
I20260620 18:43:00.904017 9222 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
I20260620 18:43:00.907755 9288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:43:00.914093 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 9222
I20260620 18:43:00.914249 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980676253068-24012-0/minicluster-data/master-0/wal/instance
I20260620 18:43:00.914680 24012 external_mini_cluster.cc:1428] Setting key b6c7486b8e337e61a3cddbfd81e63dd2
I20260620 18:43:00.919237 9288 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1e74114c0814dac9c07d59a4ecf8657: Bootstrap starting.
I20260620 18:43:00.922336 9288 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e1e74114c0814dac9c07d59a4ecf8657: Neither blocks nor log segments found. Creating new log.
I20260620 18:43:00.923470 9288 log.cc:824] T 00000000000000000000000000000000 P e1e74114c0814dac9c07d59a4ecf8657: Log is configured to *not* fsync() on all Append() calls
I20260620 18:43:00.926920 9288 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1e74114c0814dac9c07d59a4ecf8657: No bootstrap required, opened a new log
I20260620 18:43:00.928893 9291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:00.916654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38290 (local address 127.23.115.62:40637)
0620 18:43:00.917689 (+ 1035us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:00.917707 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:00.917750 (+ 43us) server_negotiation.cc:407] Connection header received
0620 18:43:00.919527 (+ 1777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:00.919552 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:00.919994 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:00.920390 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:00.920961 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:00.922204 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:00.923612 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:00.924800 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:00.924991 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:00.926236 (+ 1245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:00.926245 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:00.926259 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:43:00.926784 (+ 525us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:00.926861 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:43:00.926883 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:00.927041 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:00.927211 (+ 170us) server_negotiation.cc:299] Negotiation successful
0620 18:43:00.927764 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":394,"threads_started":1}
I20260620 18:43:00.935074 9288 raft_consensus.cc:348] T 00000000000000000000000000000000 P e1e74114c0814dac9c07d59a4ecf8657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1e74114c0814dac9c07d59a4ecf8657" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 40637 } }
I20260620 18:43:00.935425 9288 raft_consensus.cc:374] T 00000000000000000000000000000000 P e1e74114c0814dac9c07d59a4ecf8657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:43:00.935524 9288 raft_consensus.cc:729] T 00000000000000000000000000000000 P e1e74114c0814dac9c07d59a4ecf8657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1e74114c0814dac9c07d59a4ecf8657, State: Initialized, Role: FOLLOWER
I20260620 18:43:00.936018 9288 consensus_queue.cc:260] T 00000000000000000000000000000000 P e1e74114c0814dac9c07d59a4ecf8657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1e74114c0814dac9c07d59a4ecf8657" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 40637 } }
I20260620 18:43:00.936240 9288 raft_consensus.cc:388] T 00000000000000000000000000000000 P e1e74114c0814dac9c07d59a4ecf8657 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:43:00.936355 9288 raft_consensus.cc:482] T 00000000000000000000000000000000 P e1e74114c0814dac9c07d59a4ecf8657 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:43:00.936511 9288 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e1e74114c0814dac9c07d59a4ecf8657 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:43:00.938784 9288 raft_consensus.cc:504] T 00000000000000000000000000000000 P e1e74114c0814dac9c07d59a4ecf8657 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1e74114c0814dac9c07d59a4ecf8657" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 40637 } }
I20260620 18:43:00.939276 9288 leader_election.cc:302] T 00000000000000000000000000000000 P e1e74114c0814dac9c07d59a4ecf8657 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e1e74114c0814dac9c07d59a4ecf8657; no voters:
I20260620 18:43:00.939834 9288 leader_election.cc:288] T 00000000000000000000000000000000 P e1e74114c0814dac9c07d59a4ecf8657 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:43:00.940006 9293 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e1e74114c0814dac9c07d59a4ecf8657 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:43:00.940907 9293 raft_consensus.cc:686] T 00000000000000000000000000000000 P e1e74114c0814dac9c07d59a4ecf8657 [term 1 LEADER]: Becoming Leader. State: Replica: e1e74114c0814dac9c07d59a4ecf8657, State: Running, Role: LEADER
I20260620 18:43:00.941377 9293 consensus_queue.cc:237] T 00000000000000000000000000000000 P e1e74114c0814dac9c07d59a4ecf8657 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1e74114c0814dac9c07d59a4ecf8657" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 40637 } }
I20260620 18:43:00.941802 9288 sys_catalog.cc:564] T 00000000000000000000000000000000 P e1e74114c0814dac9c07d59a4ecf8657 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:43:00.943977 9295 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1e74114c0814dac9c07d59a4ecf8657 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1e74114c0814dac9c07d59a4ecf8657. Latest consensus state: current_term: 1 leader_uuid: "e1e74114c0814dac9c07d59a4ecf8657" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1e74114c0814dac9c07d59a4ecf8657" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 40637 } } }
I20260620 18:43:00.944003 9294 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1e74114c0814dac9c07d59a4ecf8657 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e1e74114c0814dac9c07d59a4ecf8657" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1e74114c0814dac9c07d59a4ecf8657" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 40637 } } }
I20260620 18:43:00.944466 9295 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1e74114c0814dac9c07d59a4ecf8657 [sys.catalog]: This master's current role is: LEADER
I20260620 18:43:00.944470 9294 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1e74114c0814dac9c07d59a4ecf8657 [sys.catalog]: This master's current role is: LEADER
I20260620 18:43:00.947358 9298 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:43:00.951503 9298 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:43:00.960557 9298 catalog_manager.cc:1347] Generated new cluster ID: d2c5c4ecc29a4f7b8d020430d7ed714e
I20260620 18:43:00.960712 9298 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:43:00.982197 9298 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:43:00.984144 9298 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:43:01.007771 9298 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e1e74114c0814dac9c07d59a4ecf8657: Generated new TSK 0
I20260620 18:43:01.008991 9298 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:43:01.028780 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980676253068-24012-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980676253068-24012-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980676253068-24012-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980676253068-24012-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.115.1:0
--local_ip_for_outbound_sockets=127.23.115.1
--webserver_interface=127.23.115.1
--webserver_port=0
--tserver_master_addrs=127.23.115.62:40637
--builtin_ntp_servers=127.23.115.20:35325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 18:43:01.363689 9312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:43:01.363958 9312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:43:01.364075 9312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:43:01.371353 9312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:43:01.371459 9312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:43:01.371598 9312 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.115.1
I20260620 18:43:01.380630 9312 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:35325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980676253068-24012-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980676253068-24012-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.115.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980676253068-24012-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.115.1
--webserver_port=0
--tserver_master_addrs=127.23.115.62:40637
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.115.1
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980676253068-24012-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:43:01.381888 9312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:43:01.383654 9312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:43:01.396173 9320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:01.396260 9317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:01.396239 9318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:01.396885 9312 server_base.cc:1034] running on GCE node
I20260620 18:43:01.397480 9312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:43:01.398412 9312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:43:01.399648 9312 hybrid_clock.cc:648] HybridClock initialized: now 1781980981399596 us; error 39 us; skew 500 ppm
I20260620 18:43:01.399971 9312 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:43:01.402478 9312 webserver.cc:460] Webserver started at http://127.23.115.1:41921/ using document root <none> and password file <none>
I20260620 18:43:01.403216 9312 fs_manager.cc:359] Metadata directory not provided
I20260620 18:43:01.403308 9312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:43:01.403586 9312 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:43:01.406437 9312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980676253068-24012-0/minicluster-data/ts-0/data/instance:
uuid: "8bbe232832d34c0fb30c686d25b40730"
format_stamp: "Formatted at 2026-06-20 18:43:01 on dist-test-slave-d645"
server_key: "e4f00a4848892982037b2753eecc2b80"
server_key_iv: "f9f125b32aae4182891e32b7a1bda508"
server_key_version: "encryptionkey@0"
I20260620 18:43:01.407269 9312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980676253068-24012-0/minicluster-data/ts-0/wal/instance:
uuid: "8bbe232832d34c0fb30c686d25b40730"
format_stamp: "Formatted at 2026-06-20 18:43:01 on dist-test-slave-d645"
server_key: "e4f00a4848892982037b2753eecc2b80"
server_key_iv: "f9f125b32aae4182891e32b7a1bda508"
server_key_version: "encryptionkey@0"
I20260620 18:43:01.412665 9312 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 18:43:01.416235 9326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:01.417558 9312 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:43:01.417732 9312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980676253068-24012-0/minicluster-data/ts-0/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980676253068-24012-0/minicluster-data/ts-0/wal
uuid: "8bbe232832d34c0fb30c686d25b40730"
format_stamp: "Formatted at 2026-06-20 18:43:01 on dist-test-slave-d645"
server_key: "e4f00a4848892982037b2753eecc2b80"
server_key_iv: "f9f125b32aae4182891e32b7a1bda508"
server_key_version: "encryptionkey@0"
I20260620 18:43:01.417934 9312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980676253068-24012-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980676253068-24012-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980676253068-24012-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:43:01.448081 9312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:43:01.448910 9312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:43:01.449177 9312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:43:01.450111 9312 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:43:01.451555 9312 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:43:01.451639 9312 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:01.451749 9312 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:43:01.451820 9312 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:01.482489 9312 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.1:38291
I20260620 18:43:01.482558 9438 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.1:38291 every 8 connection(s)
I20260620 18:43:01.483745 9312 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980676253068-24012-0/minicluster-data/ts-0/data/info.pb
I20260620 18:43:01.485306 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 9312
I20260620 18:43:01.485443 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980676253068-24012-0/minicluster-data/ts-0/wal/instance
I20260620 18:43:01.485847 24012 external_mini_cluster.cc:1428] Setting key ceda206262a303a829510d79c4e601aa
I20260620 18:43:01.489116 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980676253068-24012-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980676253068-24012-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980676253068-24012-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980676253068-24012-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.115.2:0
--local_ip_for_outbound_sockets=127.23.115.2
--webserver_interface=127.23.115.2
--webserver_port=0
--tserver_master_addrs=127.23.115.62:40637
--builtin_ntp_servers=127.23.115.20:35325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 18:43:01.498418 9441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:01.487676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.1:55743 (local address 127.23.115.62:40637)
0620 18:43:01.488089 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:01.488096 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:01.489496 (+ 1400us) server_negotiation.cc:407] Connection header received
0620 18:43:01.491323 (+ 1827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:01.491329 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:01.491404 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:01.491515 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:01.492501 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:01.493490 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:01.495107 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:01.495673 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:01.495807 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:01.497343 (+ 1536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:01.497349 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:01.497353 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:43:01.497375 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:01.497423 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:43:01.497437 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:01.497523 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:01.498003 (+ 480us) server_negotiation.cc:299] Negotiation successful
0620 18:43:01.498100 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":200,"threads_started":1}
I20260620 18:43:01.500017 9442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:01.488323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:40637 (local address 127.23.115.1:55743)
0620 18:43:01.489345 (+ 1022us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:01.489388 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:01.491076 (+ 1688us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:01.491702 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:01.491710 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:01.491795 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:43:01.492343 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:01.492365 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:01.493671 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:01.493675 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:01.494927 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:01.494970 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:01.495947 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:01.495955 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:01.496254 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:01.497031 (+ 777us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:43:01.497034 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:43:01.497077 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:43:01.497165 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:43:01.497646 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:43:01.497680 (+ 34us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:43:01.497691 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 18:43:01.497980 (+ 289us) client_negotiation.cc:240] Negotiation successful
0620 18:43:01.498646 (+ 666us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":328,"threads_started":1}
I20260620 18:43:01.501627 9439 heartbeater.cc:347] Connected to a master server at 127.23.115.62:40637
I20260620 18:43:01.501963 9439 heartbeater.cc:464] Registering TS with master...
I20260620 18:43:01.502862 9439 heartbeater.cc:511] Master 127.23.115.62:40637 requested a full tablet report, sending...
I20260620 18:43:01.505012 9253 ts_manager.cc:194] Registered new tserver with Master: 8bbe232832d34c0fb30c686d25b40730 (127.23.115.1:38291)
I20260620 18:43:01.506546 9253 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.115.1:55743
W20260620 18:43:01.812404 9443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:43:01.812805 9443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:43:01.812951 9443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:43:01.820142 9443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:43:01.820250 9443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:43:01.820391 9443 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.115.2
I20260620 18:43:01.829526 9443 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:35325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980676253068-24012-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980676253068-24012-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.115.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980676253068-24012-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.115.2
--webserver_port=0
--tserver_master_addrs=127.23.115.62:40637
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.115.2
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980676253068-24012-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:43:01.830789 9443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:43:01.832504 9443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:43:01.844599 9448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:01.844568 9449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:01.844663 9451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:01.846263 9443 server_base.cc:1034] running on GCE node
I20260620 18:43:01.846848 9443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:43:01.847868 9443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:43:01.849071 9443 hybrid_clock.cc:648] HybridClock initialized: now 1781980981849016 us; error 43 us; skew 500 ppm
I20260620 18:43:01.849388 9443 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:43:01.851610 9443 webserver.cc:460] Webserver started at http://127.23.115.2:44717/ using document root <none> and password file <none>
I20260620 18:43:01.852253 9443 fs_manager.cc:359] Metadata directory not provided
I20260620 18:43:01.852339 9443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:43:01.852615 9443 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:43:01.855784 9443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980676253068-24012-0/minicluster-data/ts-1/data/instance:
uuid: "697d0c986c874a3198deef9572543c12"
format_stamp: "Formatted at 2026-06-20 18:43:01 on dist-test-slave-d645"
server_key: "f0e9efa967167cab504a23a9a6b0b41b"
server_key_iv: "a176d41e93e64fe7c681372120d0b048"
server_key_version: "encryptionkey@0"
I20260620 18:43:01.856523 9443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980676253068-24012-0/minicluster-data/ts-1/wal/instance:
uuid: "697d0c986c874a3198deef9572543c12"
format_stamp: "Formatted at 2026-06-20 18:43:01 on dist-test-slave-d645"
server_key: "f0e9efa967167cab504a23a9a6b0b41b"
server_key_iv: "a176d41e93e64fe7c681372120d0b048"
server_key_version: "encryptionkey@0"
I20260620 18:43:01.861536 9443 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 18:43:01.864935 9457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:01.866273 9443 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260620 18:43:01.866431 9443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980676253068-24012-0/minicluster-data/ts-1/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980676253068-24012-0/minicluster-data/ts-1/wal
uuid: "697d0c986c874a3198deef9572543c12"
format_stamp: "Formatted at 2026-06-20 18:43:01 on dist-test-slave-d645"
server_key: "f0e9efa967167cab504a23a9a6b0b41b"
server_key_iv: "a176d41e93e64fe7c681372120d0b048"
server_key_version: "encryptionkey@0"
I20260620 18:43:01.866633 9443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980676253068-24012-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980676253068-24012-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980676253068-24012-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:43:01.885143 9443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:43:01.885985 9443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:43:01.886276 9443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:43:01.887317 9443 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:43:01.888770 9443 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:43:01.888870 9443 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:01.888978 9443 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:43:01.889055 9443 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:01.920907 9443 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.2:33405
I20260620 18:43:01.920943 9569 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.2:33405 every 8 connection(s)
I20260620 18:43:01.922333 9443 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980676253068-24012-0/minicluster-data/ts-1/data/info.pb
I20260620 18:43:01.924841 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 9443
I20260620 18:43:01.925027 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980676253068-24012-0/minicluster-data/ts-1/wal/instance
I20260620 18:43:01.925498 24012 external_mini_cluster.cc:1428] Setting key dac3c5834d3c56817a6009838c9a9e31
I20260620 18:43:01.928650 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980676253068-24012-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980676253068-24012-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980676253068-24012-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980676253068-24012-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.115.3:0
--local_ip_for_outbound_sockets=127.23.115.3
--webserver_interface=127.23.115.3
--webserver_port=0
--tserver_master_addrs=127.23.115.62:40637
--builtin_ntp_servers=127.23.115.20:35325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 18:43:01.936499 9441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:01.925104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.2:39597 (local address 127.23.115.62:40637)
0620 18:43:01.925970 (+ 866us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:01.925975 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:01.926773 (+ 798us) server_negotiation.cc:407] Connection header received
0620 18:43:01.929046 (+ 2273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:01.929050 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:01.929123 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:01.929210 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:01.930573 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:01.931408 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:01.933461 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:01.933862 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:01.933965 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:01.935717 (+ 1752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:01.935722 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:01.935725 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:43:01.935745 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:01.935778 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:43:01.935789 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:01.935863 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:01.936164 (+ 301us) server_negotiation.cc:299] Negotiation successful
0620 18:43:01.936269 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":749}
I20260620 18:43:01.937772 9572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:01.925562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:40637 (local address 127.23.115.2:39597)
0620 18:43:01.926623 (+ 1061us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:01.926666 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:01.928755 (+ 2089us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:01.929467 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:01.929479 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:01.929598 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:43:01.930385 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:01.930409 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:01.931568 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:01.931573 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:01.933276 (+ 1703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:01.933291 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:01.934134 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:01.934143 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:01.934457 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:01.935329 (+ 872us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:43:01.935332 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:43:01.935384 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:43:01.935504 (+ 120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:43:01.935881 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:43:01.935891 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:43:01.935898 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 18:43:01.936095 (+ 197us) client_negotiation.cc:240] Negotiation successful
0620 18:43:01.936521 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":316,"threads_started":1}
I20260620 18:43:01.939337 9570 heartbeater.cc:347] Connected to a master server at 127.23.115.62:40637
I20260620 18:43:01.939647 9570 heartbeater.cc:464] Registering TS with master...
I20260620 18:43:01.940433 9570 heartbeater.cc:511] Master 127.23.115.62:40637 requested a full tablet report, sending...
I20260620 18:43:01.941908 9253 ts_manager.cc:194] Registered new tserver with Master: 697d0c986c874a3198deef9572543c12 (127.23.115.2:33405)
I20260620 18:43:01.942979 9253 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.115.2:39597
W20260620 18:43:02.245675 9573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:43:02.245981 9573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:43:02.246066 9573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:43:02.253293 9573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:43:02.253396 9573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:43:02.253521 9573 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.115.3
I20260620 18:43:02.262704 9573 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:35325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980676253068-24012-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980676253068-24012-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.115.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980676253068-24012-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.115.3
--webserver_port=0
--tserver_master_addrs=127.23.115.62:40637
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.115.3
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980676253068-24012-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:43:02.264001 9573 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:43:02.265767 9573 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:43:02.279378 9579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:02.279412 9578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:02.279333 9581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:02.280761 9573 server_base.cc:1034] running on GCE node
I20260620 18:43:02.281375 9573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:43:02.282370 9573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:43:02.283603 9573 hybrid_clock.cc:648] HybridClock initialized: now 1781980982283551 us; error 41 us; skew 500 ppm
I20260620 18:43:02.283954 9573 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:43:02.286273 9573 webserver.cc:460] Webserver started at http://127.23.115.3:34417/ using document root <none> and password file <none>
I20260620 18:43:02.286991 9573 fs_manager.cc:359] Metadata directory not provided
I20260620 18:43:02.287088 9573 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:43:02.287389 9573 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:43:02.290206 9573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980676253068-24012-0/minicluster-data/ts-2/data/instance:
uuid: "18fda0aa5eb24b54b3ce44e3e7031e33"
format_stamp: "Formatted at 2026-06-20 18:43:02 on dist-test-slave-d645"
server_key: "c8901eafdaa867be049874b5b261355b"
server_key_iv: "777bd957fa9d3f393968bf456bcbd4bf"
server_key_version: "encryptionkey@0"
I20260620 18:43:02.290999 9573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980676253068-24012-0/minicluster-data/ts-2/wal/instance:
uuid: "18fda0aa5eb24b54b3ce44e3e7031e33"
format_stamp: "Formatted at 2026-06-20 18:43:02 on dist-test-slave-d645"
server_key: "c8901eafdaa867be049874b5b261355b"
server_key_iv: "777bd957fa9d3f393968bf456bcbd4bf"
server_key_version: "encryptionkey@0"
I20260620 18:43:02.296191 9573 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 18:43:02.299805 9587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:02.301112 9573 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:43:02.301306 9573 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980676253068-24012-0/minicluster-data/ts-2/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980676253068-24012-0/minicluster-data/ts-2/wal
uuid: "18fda0aa5eb24b54b3ce44e3e7031e33"
format_stamp: "Formatted at 2026-06-20 18:43:02 on dist-test-slave-d645"
server_key: "c8901eafdaa867be049874b5b261355b"
server_key_iv: "777bd957fa9d3f393968bf456bcbd4bf"
server_key_version: "encryptionkey@0"
I20260620 18:43:02.301509 9573 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980676253068-24012-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980676253068-24012-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980676253068-24012-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:43:02.318615 9573 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:43:02.319576 9573 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:43:02.319855 9573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:43:02.320825 9573 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:43:02.322263 9573 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:43:02.322359 9573 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:02.322458 9573 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:43:02.322525 9573 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:02.362839 9573 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.3:40051
I20260620 18:43:02.362885 9699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.3:40051 every 8 connection(s)
I20260620 18:43:02.364220 9573 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980676253068-24012-0/minicluster-data/ts-2/data/info.pb
I20260620 18:43:02.365893 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 9573
I20260620 18:43:02.366053 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980676253068-24012-0/minicluster-data/ts-2/wal/instance
I20260620 18:43:02.370720 24012 external_mini_cluster.cc:1428] Setting key e2ba3485f0824d942eb25e9f984b1f71
I20260620 18:43:02.377133 9441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:02.367070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.3:46871 (local address 127.23.115.62:40637)
0620 18:43:02.367224 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:02.367229 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:02.368894 (+ 1665us) server_negotiation.cc:407] Connection header received
0620 18:43:02.370717 (+ 1823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:02.370723 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:02.370807 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:02.370926 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:02.372089 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:02.373067 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:02.374560 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:02.374988 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:02.375097 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:02.376354 (+ 1257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:02.376360 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:02.376364 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:43:02.376384 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:02.376414 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:43:02.376426 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:02.376498 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:02.376691 (+ 193us) server_negotiation.cc:299] Negotiation successful
0620 18:43:02.376793 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 18:43:02.378060 9702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:02.367819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:40637 (local address 127.23.115.3:46871)
0620 18:43:02.368742 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:02.368776 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:02.370441 (+ 1665us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:02.371160 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:02.371168 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:02.371261 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:43:02.371943 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:02.371963 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:02.373199 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:02.373211 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:02.374427 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:02.374440 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:02.375187 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:02.375194 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:02.375420 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:02.376046 (+ 626us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:43:02.376048 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:43:02.376093 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:43:02.376168 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:43:02.376507 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:43:02.376517 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:43:02.376525 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 18:43:02.376672 (+ 147us) client_negotiation.cc:240] Negotiation successful
0620 18:43:02.377063 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":300,"threads_started":1}
I20260620 18:43:02.379334 9700 heartbeater.cc:347] Connected to a master server at 127.23.115.62:40637
I20260620 18:43:02.379580 9700 heartbeater.cc:464] Registering TS with master...
I20260620 18:43:02.380259 9700 heartbeater.cc:511] Master 127.23.115.62:40637 requested a full tablet report, sending...
I20260620 18:43:02.381538 9253 ts_manager.cc:194] Registered new tserver with Master: 18fda0aa5eb24b54b3ce44e3e7031e33 (127.23.115.3:40051)
I20260620 18:43:02.382380 9253 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.115.3:46871
I20260620 18:43:02.385491 24012 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:43:02.395273 9441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:02.389474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38296 (local address 127.23.115.62:40637)
0620 18:43:02.389677 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:02.389683 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:02.389979 (+ 296us) server_negotiation.cc:407] Connection header received
0620 18:43:02.390178 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:02.390182 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:02.390270 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:02.390369 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:02.390839 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:02.391639 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:02.393040 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:02.393464 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:02.393564 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:02.394519 (+ 955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:02.394523 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:02.394526 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:43:02.394559 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:02.394593 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:43:02.394605 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:02.394675 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:02.394858 (+ 183us) server_negotiation.cc:299] Negotiation successful
0620 18:43:02.394995 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 18:43:02.399286 9253 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38296:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:43:02.405987 9253 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:43:02.430625 9716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:02.420360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.3:40051 (local address 127.0.0.1:45370)
0620 18:43:02.421314 (+ 954us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:02.421333 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:02.421447 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:02.422642 (+ 1195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:02.422649 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:02.422678 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:43:02.423072 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:02.423083 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:02.424641 (+ 1558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:02.424646 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:02.427762 (+ 3116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:02.427777 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:02.429613 (+ 1836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:02.429619 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:02.429845 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:02.429901 (+ 56us) client_negotiation.cc:769] Sending connection context
0620 18:43:02.430019 (+ 118us) client_negotiation.cc:240] Negotiation successful
0620 18:43:02.430237 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":833,"mutex_wait_us":2,"thread_start_us":711,"threads_started":1}
I20260620 18:43:02.431408 9717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:02.420889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45370 (local address 127.23.115.3:40051)
0620 18:43:02.421976 (+ 1087us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:02.421984 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:02.422015 (+ 31us) server_negotiation.cc:407] Connection header received
0620 18:43:02.422116 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:02.422123 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:02.422405 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:02.423247 (+ 842us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:43:02.423435 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:02.424510 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:02.427974 (+ 3464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:02.429398 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:02.429536 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:02.429799 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:02.430889 (+ 1090us) server_negotiation.cc:299] Negotiation successful
0620 18:43:02.431079 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":922,"thread_start_us":792,"threads_started":1}
I20260620 18:43:02.432489 9712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:02.417866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.2:33405 (local address 127.0.0.1:60992)
0620 18:43:02.418672 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:02.418737 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:02.418978 (+ 241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:02.425895 (+ 6917us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:02.425899 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:02.425927 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:43:02.426368 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:02.426381 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:02.428152 (+ 1771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:02.428158 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:02.429596 (+ 1438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:02.429608 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:02.431634 (+ 2026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:02.431640 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:02.431918 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:02.431952 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 18:43:02.432017 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 18:43:02.432139 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"mutex_wait_us":27,"thread_start_us":617,"threads_started":1}
I20260620 18:43:02.433121 9715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:02.418306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60964 (local address 127.23.115.1:38291)
0620 18:43:02.425006 (+ 6700us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:02.425014 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:02.425040 (+ 26us) server_negotiation.cc:407] Connection header received
0620 18:43:02.425136 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:02.425144 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:02.425376 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:02.426285 (+ 909us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:43:02.426514 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:02.427630 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:02.430345 (+ 2715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:02.431678 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:02.431951 (+ 273us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:02.432175 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:02.432590 (+ 415us) server_negotiation.cc:299] Negotiation successful
0620 18:43:02.432776 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6550,"thread_start_us":6441,"threads_started":1}
I20260620 18:43:02.433135 9714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:02.418180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60992 (local address 127.23.115.2:33405)
0620 18:43:02.425325 (+ 7145us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:02.425330 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:02.425350 (+ 20us) server_negotiation.cc:407] Connection header received
0620 18:43:02.425440 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:02.425446 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:02.425648 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:02.425895 (+ 247us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:43:02.426570 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:02.428035 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:02.429829 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:02.431436 (+ 1607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:02.432315 (+ 879us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:02.432567 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:02.432628 (+ 61us) server_negotiation.cc:299] Negotiation successful
0620 18:43:02.432775 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6967,"thread_start_us":6849,"threads_started":1}
I20260620 18:43:02.432986 9713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:02.418037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.1:38291 (local address 127.0.0.1:60964)
0620 18:43:02.420303 (+ 2266us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:02.420319 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:02.420429 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:02.425580 (+ 5151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:02.425585 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:02.425615 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:43:02.426074 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:02.426087 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:02.427765 (+ 1678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:02.427771 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:02.430121 (+ 2350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:02.430134 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:02.431992 (+ 1858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:02.431999 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:02.432170 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:02.432201 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 18:43:02.432284 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 18:43:02.432417 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2112,"mutex_wait_us":171,"thread_start_us":2717,"threads_started":1}
I20260620 18:43:02.435425 9635 tablet_service.cc:1467] Processing CreateTablet for tablet 7749b1ff1584458a84fc1d9e3ed57207 (DEFAULT_TABLE table=test-table [id=d240c0d53fd144e39b26ae41106b7a24]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:43:02.436498 9505 tablet_service.cc:1467] Processing CreateTablet for tablet 7749b1ff1584458a84fc1d9e3ed57207 (DEFAULT_TABLE table=test-table [id=d240c0d53fd144e39b26ae41106b7a24]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:43:02.436969 9635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7749b1ff1584458a84fc1d9e3ed57207. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:43:02.436655 9374 tablet_service.cc:1467] Processing CreateTablet for tablet 7749b1ff1584458a84fc1d9e3ed57207 (DEFAULT_TABLE table=test-table [id=d240c0d53fd144e39b26ae41106b7a24]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:43:02.437556 9505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7749b1ff1584458a84fc1d9e3ed57207. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:43:02.437799 9374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7749b1ff1584458a84fc1d9e3ed57207. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:43:02.449692 9718 tablet_bootstrap.cc:492] T 7749b1ff1584458a84fc1d9e3ed57207 P 697d0c986c874a3198deef9572543c12: Bootstrap starting.
I20260620 18:43:02.451035 9719 tablet_bootstrap.cc:492] T 7749b1ff1584458a84fc1d9e3ed57207 P 18fda0aa5eb24b54b3ce44e3e7031e33: Bootstrap starting.
I20260620 18:43:02.452220 9720 tablet_bootstrap.cc:492] T 7749b1ff1584458a84fc1d9e3ed57207 P 8bbe232832d34c0fb30c686d25b40730: Bootstrap starting.
I20260620 18:43:02.454515 9718 tablet_bootstrap.cc:654] T 7749b1ff1584458a84fc1d9e3ed57207 P 697d0c986c874a3198deef9572543c12: Neither blocks nor log segments found. Creating new log.
I20260620 18:43:02.454523 9719 tablet_bootstrap.cc:654] T 7749b1ff1584458a84fc1d9e3ed57207 P 18fda0aa5eb24b54b3ce44e3e7031e33: Neither blocks nor log segments found. Creating new log.
I20260620 18:43:02.454864 9720 tablet_bootstrap.cc:654] T 7749b1ff1584458a84fc1d9e3ed57207 P 8bbe232832d34c0fb30c686d25b40730: Neither blocks nor log segments found. Creating new log.
I20260620 18:43:02.455818 9719 log.cc:824] T 7749b1ff1584458a84fc1d9e3ed57207 P 18fda0aa5eb24b54b3ce44e3e7031e33: Log is configured to *not* fsync() on all Append() calls
I20260620 18:43:02.455818 9718 log.cc:824] T 7749b1ff1584458a84fc1d9e3ed57207 P 697d0c986c874a3198deef9572543c12: Log is configured to *not* fsync() on all Append() calls
I20260620 18:43:02.455890 9720 log.cc:824] T 7749b1ff1584458a84fc1d9e3ed57207 P 8bbe232832d34c0fb30c686d25b40730: Log is configured to *not* fsync() on all Append() calls
I20260620 18:43:02.459724 9718 tablet_bootstrap.cc:492] T 7749b1ff1584458a84fc1d9e3ed57207 P 697d0c986c874a3198deef9572543c12: No bootstrap required, opened a new log
I20260620 18:43:02.460072 9718 ts_tablet_manager.cc:1397] T 7749b1ff1584458a84fc1d9e3ed57207 P 697d0c986c874a3198deef9572543c12: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.003s
I20260620 18:43:02.465060 9719 tablet_bootstrap.cc:492] T 7749b1ff1584458a84fc1d9e3ed57207 P 18fda0aa5eb24b54b3ce44e3e7031e33: No bootstrap required, opened a new log
I20260620 18:43:02.465252 9720 tablet_bootstrap.cc:492] T 7749b1ff1584458a84fc1d9e3ed57207 P 8bbe232832d34c0fb30c686d25b40730: No bootstrap required, opened a new log
I20260620 18:43:02.465359 9719 ts_tablet_manager.cc:1397] T 7749b1ff1584458a84fc1d9e3ed57207 P 18fda0aa5eb24b54b3ce44e3e7031e33: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.006s
I20260620 18:43:02.465520 9720 ts_tablet_manager.cc:1397] T 7749b1ff1584458a84fc1d9e3ed57207 P 8bbe232832d34c0fb30c686d25b40730: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.000s
I20260620 18:43:02.471237 9718 raft_consensus.cc:348] T 7749b1ff1584458a84fc1d9e3ed57207 P 697d0c986c874a3198deef9572543c12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "697d0c986c874a3198deef9572543c12" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 33405 } } peers { permanent_uuid: "8bbe232832d34c0fb30c686d25b40730" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 38291 } } peers { permanent_uuid: "18fda0aa5eb24b54b3ce44e3e7031e33" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 40051 } }
I20260620 18:43:02.471735 9718 raft_consensus.cc:374] T 7749b1ff1584458a84fc1d9e3ed57207 P 697d0c986c874a3198deef9572543c12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:43:02.471880 9718 raft_consensus.cc:729] T 7749b1ff1584458a84fc1d9e3ed57207 P 697d0c986c874a3198deef9572543c12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 697d0c986c874a3198deef9572543c12, State: Initialized, Role: FOLLOWER
I20260620 18:43:02.472574 9718 consensus_queue.cc:260] T 7749b1ff1584458a84fc1d9e3ed57207 P 697d0c986c874a3198deef9572543c12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "697d0c986c874a3198deef9572543c12" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 33405 } } peers { permanent_uuid: "8bbe232832d34c0fb30c686d25b40730" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 38291 } } peers { permanent_uuid: "18fda0aa5eb24b54b3ce44e3e7031e33" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 40051 } }
I20260620 18:43:02.472671 9719 raft_consensus.cc:348] T 7749b1ff1584458a84fc1d9e3ed57207 P 18fda0aa5eb24b54b3ce44e3e7031e33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "697d0c986c874a3198deef9572543c12" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 33405 } } peers { permanent_uuid: "8bbe232832d34c0fb30c686d25b40730" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 38291 } } peers { permanent_uuid: "18fda0aa5eb24b54b3ce44e3e7031e33" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 40051 } }
I20260620 18:43:02.474594 9719 raft_consensus.cc:374] T 7749b1ff1584458a84fc1d9e3ed57207 P 18fda0aa5eb24b54b3ce44e3e7031e33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:43:02.474861 9719 raft_consensus.cc:729] T 7749b1ff1584458a84fc1d9e3ed57207 P 18fda0aa5eb24b54b3ce44e3e7031e33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18fda0aa5eb24b54b3ce44e3e7031e33, State: Initialized, Role: FOLLOWER
I20260620 18:43:02.475725 9719 consensus_queue.cc:260] T 7749b1ff1584458a84fc1d9e3ed57207 P 18fda0aa5eb24b54b3ce44e3e7031e33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "697d0c986c874a3198deef9572543c12" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 33405 } } peers { permanent_uuid: "8bbe232832d34c0fb30c686d25b40730" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 38291 } } peers { permanent_uuid: "18fda0aa5eb24b54b3ce44e3e7031e33" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 40051 } }
I20260620 18:43:02.476181 9720 raft_consensus.cc:348] T 7749b1ff1584458a84fc1d9e3ed57207 P 8bbe232832d34c0fb30c686d25b40730 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "697d0c986c874a3198deef9572543c12" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 33405 } } peers { permanent_uuid: "8bbe232832d34c0fb30c686d25b40730" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 38291 } } peers { permanent_uuid: "18fda0aa5eb24b54b3ce44e3e7031e33" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 40051 } }
I20260620 18:43:02.476761 9720 raft_consensus.cc:374] T 7749b1ff1584458a84fc1d9e3ed57207 P 8bbe232832d34c0fb30c686d25b40730 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:43:02.476843 9700 heartbeater.cc:503] Master 127.23.115.62:40637 was elected leader, sending a full tablet report...
I20260620 18:43:02.476907 9720 raft_consensus.cc:729] T 7749b1ff1584458a84fc1d9e3ed57207 P 8bbe232832d34c0fb30c686d25b40730 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bbe232832d34c0fb30c686d25b40730, State: Initialized, Role: FOLLOWER
I20260620 18:43:02.477542 9719 ts_tablet_manager.cc:1428] T 7749b1ff1584458a84fc1d9e3ed57207 P 18fda0aa5eb24b54b3ce44e3e7031e33: Time spent starting tablet: real 0.012s user 0.011s sys 0.001s
I20260620 18:43:02.477658 9720 consensus_queue.cc:260] T 7749b1ff1584458a84fc1d9e3ed57207 P 8bbe232832d34c0fb30c686d25b40730 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "697d0c986c874a3198deef9572543c12" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 33405 } } peers { permanent_uuid: "8bbe232832d34c0fb30c686d25b40730" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 38291 } } peers { permanent_uuid: "18fda0aa5eb24b54b3ce44e3e7031e33" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 40051 } }
I20260620 18:43:02.478857 9570 heartbeater.cc:503] Master 127.23.115.62:40637 was elected leader, sending a full tablet report...
I20260620 18:43:02.479507 9718 ts_tablet_manager.cc:1428] T 7749b1ff1584458a84fc1d9e3ed57207 P 697d0c986c874a3198deef9572543c12: Time spent starting tablet: real 0.019s user 0.014s sys 0.006s
I20260620 18:43:02.480420 9439 heartbeater.cc:503] Master 127.23.115.62:40637 was elected leader, sending a full tablet report...
I20260620 18:43:02.482127 9720 ts_tablet_manager.cc:1428] T 7749b1ff1584458a84fc1d9e3ed57207 P 8bbe232832d34c0fb30c686d25b40730: Time spent starting tablet: real 0.016s user 0.004s sys 0.012s
W20260620 18:43:02.485817 9440 tablet.cc:2369] T 7749b1ff1584458a84fc1d9e3ed57207 P 8bbe232832d34c0fb30c686d25b40730: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:43:02.616271 9701 tablet.cc:2369] T 7749b1ff1584458a84fc1d9e3ed57207 P 18fda0aa5eb24b54b3ce44e3e7031e33: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:43:02.616979 9724 raft_consensus.cc:482] T 7749b1ff1584458a84fc1d9e3ed57207 P 697d0c986c874a3198deef9572543c12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:43:02.617225 9724 raft_consensus.cc:504] T 7749b1ff1584458a84fc1d9e3ed57207 P 697d0c986c874a3198deef9572543c12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "697d0c986c874a3198deef9572543c12" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 33405 } } peers { permanent_uuid: "8bbe232832d34c0fb30c686d25b40730" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 38291 } } peers { permanent_uuid: "18fda0aa5eb24b54b3ce44e3e7031e33" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 40051 } }
I20260620 18:43:02.618891 9724 leader_election.cc:288] T 7749b1ff1584458a84fc1d9e3ed57207 P 697d0c986c874a3198deef9572543c12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8bbe232832d34c0fb30c686d25b40730 (127.23.115.1:38291), 18fda0aa5eb24b54b3ce44e3e7031e33 (127.23.115.3:40051)
I20260620 18:43:02.625362 9717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:02.619484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.2:42451 (local address 127.23.115.3:40051)
0620 18:43:02.619662 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:02.619668 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:02.620210 (+ 542us) server_negotiation.cc:407] Connection header received
0620 18:43:02.620371 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:02.620375 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:02.620433 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:02.620502 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:43:02.621108 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:02.621838 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:02.623788 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:02.624542 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:02.624695 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:02.624761 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:02.625017 (+ 256us) server_negotiation.cc:299] Negotiation successful
0620 18:43:02.625136 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 18:43:02.625366 9715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:02.619394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.2:54045 (local address 127.23.115.1:38291)
0620 18:43:02.619570 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:02.619576 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:02.620023 (+ 447us) server_negotiation.cc:407] Connection header received
0620 18:43:02.620214 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:02.620218 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:02.620285 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:02.620368 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:43:02.620989 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:02.621742 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:02.623774 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:02.624507 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:02.624651 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:02.624732 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:02.625000 (+ 268us) server_negotiation.cc:299] Negotiation successful
0620 18:43:02.625120 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 18:43:02.625499 9727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:02.619144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.1:38291 (local address 127.23.115.2:54045)
0620 18:43:02.619917 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:02.619934 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:02.620081 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:02.620397 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:02.620400 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:02.620423 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:43:02.620851 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:02.620860 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:02.621857 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:02.621863 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:02.623576 (+ 1713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:02.623588 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:02.624708 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:02.624714 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:02.624856 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:02.624880 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 18:43:02.624956 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 18:43:02.625079 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"mutex_wait_us":129,"spinlock_wait_cycles":15744,"thread_start_us":438,"threads_started":1}
I20260620 18:43:02.625523 9728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:02.619316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.3:40051 (local address 127.23.115.2:42451)
0620 18:43:02.620116 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:02.620130 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:02.620251 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:02.620533 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:02.620537 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:02.620575 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:43:02.620966 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:02.620974 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:02.621963 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:02.621968 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:02.623578 (+ 1610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:02.623590 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:02.624724 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:02.624730 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:02.624878 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:02.624900 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 18:43:02.624972 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 18:43:02.625095 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":692,"mutex_wait_us":90,"thread_start_us":506,"threads_started":1}
I20260620 18:43:02.626623 9655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7749b1ff1584458a84fc1d9e3ed57207" candidate_uuid: "697d0c986c874a3198deef9572543c12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18fda0aa5eb24b54b3ce44e3e7031e33" is_pre_election: true
I20260620 18:43:02.626623 9394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7749b1ff1584458a84fc1d9e3ed57207" candidate_uuid: "697d0c986c874a3198deef9572543c12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bbe232832d34c0fb30c686d25b40730" is_pre_election: true
I20260620 18:43:02.627202 9655 raft_consensus.cc:2454] T 7749b1ff1584458a84fc1d9e3ed57207 P 18fda0aa5eb24b54b3ce44e3e7031e33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 697d0c986c874a3198deef9572543c12 in term 0.
I20260620 18:43:02.627197 9394 raft_consensus.cc:2454] T 7749b1ff1584458a84fc1d9e3ed57207 P 8bbe232832d34c0fb30c686d25b40730 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 697d0c986c874a3198deef9572543c12 in term 0.
I20260620 18:43:02.627941 9459 leader_election.cc:302] T 7749b1ff1584458a84fc1d9e3ed57207 P 697d0c986c874a3198deef9572543c12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 697d0c986c874a3198deef9572543c12, 8bbe232832d34c0fb30c686d25b40730; no voters:
I20260620 18:43:02.628351 9724 raft_consensus.cc:2781] T 7749b1ff1584458a84fc1d9e3ed57207 P 697d0c986c874a3198deef9572543c12 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:43:02.628499 9724 raft_consensus.cc:482] T 7749b1ff1584458a84fc1d9e3ed57207 P 697d0c986c874a3198deef9572543c12 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:43:02.628623 9724 raft_consensus.cc:3037] T 7749b1ff1584458a84fc1d9e3ed57207 P 697d0c986c874a3198deef9572543c12 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:43:02.630959 9724 raft_consensus.cc:504] T 7749b1ff1584458a84fc1d9e3ed57207 P 697d0c986c874a3198deef9572543c12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "697d0c986c874a3198deef9572543c12" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 33405 } } peers { permanent_uuid: "8bbe232832d34c0fb30c686d25b40730" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 38291 } } peers { permanent_uuid: "18fda0aa5eb24b54b3ce44e3e7031e33" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 40051 } }
I20260620 18:43:02.631628 9724 leader_election.cc:288] T 7749b1ff1584458a84fc1d9e3ed57207 P 697d0c986c874a3198deef9572543c12 [CANDIDATE]: Term 1 election: Requested vote from peers 8bbe232832d34c0fb30c686d25b40730 (127.23.115.1:38291), 18fda0aa5eb24b54b3ce44e3e7031e33 (127.23.115.3:40051)
I20260620 18:43:02.632148 9655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7749b1ff1584458a84fc1d9e3ed57207" candidate_uuid: "697d0c986c874a3198deef9572543c12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18fda0aa5eb24b54b3ce44e3e7031e33"
I20260620 18:43:02.632159 9394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7749b1ff1584458a84fc1d9e3ed57207" candidate_uuid: "697d0c986c874a3198deef9572543c12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bbe232832d34c0fb30c686d25b40730"
I20260620 18:43:02.632354 9655 raft_consensus.cc:3037] T 7749b1ff1584458a84fc1d9e3ed57207 P 18fda0aa5eb24b54b3ce44e3e7031e33 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:43:02.632375 9394 raft_consensus.cc:3037] T 7749b1ff1584458a84fc1d9e3ed57207 P 8bbe232832d34c0fb30c686d25b40730 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:43:02.634466 9655 raft_consensus.cc:2454] T 7749b1ff1584458a84fc1d9e3ed57207 P 18fda0aa5eb24b54b3ce44e3e7031e33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 697d0c986c874a3198deef9572543c12 in term 1.
I20260620 18:43:02.634654 9394 raft_consensus.cc:2454] T 7749b1ff1584458a84fc1d9e3ed57207 P 8bbe232832d34c0fb30c686d25b40730 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 697d0c986c874a3198deef9572543c12 in term 1.
I20260620 18:43:02.634874 9460 leader_election.cc:302] T 7749b1ff1584458a84fc1d9e3ed57207 P 697d0c986c874a3198deef9572543c12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18fda0aa5eb24b54b3ce44e3e7031e33, 697d0c986c874a3198deef9572543c12; no voters:
I20260620 18:43:02.635219 9724 raft_consensus.cc:2781] T 7749b1ff1584458a84fc1d9e3ed57207 P 697d0c986c874a3198deef9572543c12 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:43:02.635645 9724 raft_consensus.cc:686] T 7749b1ff1584458a84fc1d9e3ed57207 P 697d0c986c874a3198deef9572543c12 [term 1 LEADER]: Becoming Leader. State: Replica: 697d0c986c874a3198deef9572543c12, State: Running, Role: LEADER
I20260620 18:43:02.636119 9724 consensus_queue.cc:237] T 7749b1ff1584458a84fc1d9e3ed57207 P 697d0c986c874a3198deef9572543c12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "697d0c986c874a3198deef9572543c12" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 33405 } } peers { permanent_uuid: "8bbe232832d34c0fb30c686d25b40730" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 38291 } } peers { permanent_uuid: "18fda0aa5eb24b54b3ce44e3e7031e33" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 40051 } }
I20260620 18:43:02.640688 9252 catalog_manager.cc:5495] T 7749b1ff1584458a84fc1d9e3ed57207 P 697d0c986c874a3198deef9572543c12 reported cstate change: term changed from 0 to 1, leader changed from <none> to 697d0c986c874a3198deef9572543c12 (127.23.115.2). New cstate: current_term: 1 leader_uuid: "697d0c986c874a3198deef9572543c12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "697d0c986c874a3198deef9572543c12" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 33405 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8bbe232832d34c0fb30c686d25b40730" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 38291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "18fda0aa5eb24b54b3ce44e3e7031e33" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 40051 } health_report { overall_health: UNKNOWN } } }
I20260620 18:43:02.674142 9714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:02.668974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32776 (local address 127.23.115.2:33405)
0620 18:43:02.669223 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:02.669229 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:02.669249 (+ 20us) server_negotiation.cc:407] Connection header received
0620 18:43:02.669330 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:02.669334 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:02.669418 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:02.669544 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:43:02.669950 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:02.670665 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:02.672112 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:02.672513 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:02.672622 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:02.673091 (+ 469us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:43:02.673586 (+ 495us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:43:02.673716 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:02.673825 (+ 109us) server_negotiation.cc:299] Negotiation successful
0620 18:43:02.673901 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
W20260620 18:43:02.674381 9571 tablet.cc:2369] T 7749b1ff1584458a84fc1d9e3ed57207 P 697d0c986c874a3198deef9572543c12: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:43:02.682257 9655 raft_consensus.cc:1261] T 7749b1ff1584458a84fc1d9e3ed57207 P 18fda0aa5eb24b54b3ce44e3e7031e33 [term 1 FOLLOWER]: Refusing update from remote peer 697d0c986c874a3198deef9572543c12: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:43:02.682273 9394 raft_consensus.cc:1261] T 7749b1ff1584458a84fc1d9e3ed57207 P 8bbe232832d34c0fb30c686d25b40730 [term 1 FOLLOWER]: Refusing update from remote peer 697d0c986c874a3198deef9572543c12: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:43:02.683180 9729 consensus_queue.cc:1035] T 7749b1ff1584458a84fc1d9e3ed57207 P 697d0c986c874a3198deef9572543c12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18fda0aa5eb24b54b3ce44e3e7031e33" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 40051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:43:02.683542 9724 consensus_queue.cc:1035] T 7749b1ff1584458a84fc1d9e3ed57207 P 697d0c986c874a3198deef9572543c12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8bbe232832d34c0fb30c686d25b40730" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:43:02.694425 9732 mvcc.cc:205] Tried to move back new op lower bound from 7298994105056952320 to 7298994104881348608. Current Snapshot: MvccSnapshot[applied={T|T < 7298994105056952320}]
I20260620 18:43:02.695622 9734 mvcc.cc:205] Tried to move back new op lower bound from 7298994105056952320 to 7298994104881348608. Current Snapshot: MvccSnapshot[applied={T|T < 7298994105056952320}]
I20260620 18:43:02.709180 9252 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38296:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:43:02.709579 9252 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38296:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:43:02.714182 9252 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e1e74114c0814dac9c07d59a4ecf8657: Sending DeleteTablet for 3 replicas of tablet 7749b1ff1584458a84fc1d9e3ed57207
I20260620 18:43:02.715358 9505 tablet_service.cc:1514] Processing DeleteTablet for tablet 7749b1ff1584458a84fc1d9e3ed57207 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:43:02 UTC) from {username='slave'} at 127.0.0.1:60992
I20260620 18:43:02.715441 9374 tablet_service.cc:1514] Processing DeleteTablet for tablet 7749b1ff1584458a84fc1d9e3ed57207 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:43:02 UTC) from {username='slave'} at 127.0.0.1:60964
I20260620 18:43:02.716235 9741 tablet_replica.cc:330] T 7749b1ff1584458a84fc1d9e3ed57207 P 697d0c986c874a3198deef9572543c12: stopping tablet replica
I20260620 18:43:02.716678 9635 tablet_service.cc:1514] Processing DeleteTablet for tablet 7749b1ff1584458a84fc1d9e3ed57207 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:43:02 UTC) from {username='slave'} at 127.0.0.1:45370
I20260620 18:43:02.716711 9741 raft_consensus.cc:2229] T 7749b1ff1584458a84fc1d9e3ed57207 P 697d0c986c874a3198deef9572543c12 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:43:02.717726 9741 raft_consensus.cc:2258] T 7749b1ff1584458a84fc1d9e3ed57207 P 697d0c986c874a3198deef9572543c12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:43:02.718842 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 9312
I20260620 18:43:02.730378 9741 ts_tablet_manager.cc:1905] T 7749b1ff1584458a84fc1d9e3ed57207 P 697d0c986c874a3198deef9572543c12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:43:02.733477 9742 tablet_replica.cc:330] T 7749b1ff1584458a84fc1d9e3ed57207 P 8bbe232832d34c0fb30c686d25b40730: stopping tablet replica
I20260620 18:43:02.733975 9742 raft_consensus.cc:2229] T 7749b1ff1584458a84fc1d9e3ed57207 P 8bbe232832d34c0fb30c686d25b40730 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:43:02.734530 9742 raft_consensus.cc:2258] T 7749b1ff1584458a84fc1d9e3ed57207 P 8bbe232832d34c0fb30c686d25b40730 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:43:02.735333 9743 tablet_replica.cc:330] T 7749b1ff1584458a84fc1d9e3ed57207 P 18fda0aa5eb24b54b3ce44e3e7031e33: stopping tablet replica
I20260620 18:43:02.735746 9743 raft_consensus.cc:2229] T 7749b1ff1584458a84fc1d9e3ed57207 P 18fda0aa5eb24b54b3ce44e3e7031e33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:43:02.736233 9743 raft_consensus.cc:2258] T 7749b1ff1584458a84fc1d9e3ed57207 P 18fda0aa5eb24b54b3ce44e3e7031e33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:43:02.739295 9741 ts_tablet_manager.cc:1918] T 7749b1ff1584458a84fc1d9e3ed57207 P 697d0c986c874a3198deef9572543c12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:43:02.739464 9741 log.cc:1196] T 7749b1ff1584458a84fc1d9e3ed57207 P 697d0c986c874a3198deef9572543c12: Deleting WAL directory at /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980676253068-24012-0/minicluster-data/ts-1/wal/wals/7749b1ff1584458a84fc1d9e3ed57207
I20260620 18:43:02.739989 9741 ts_tablet_manager.cc:1939] T 7749b1ff1584458a84fc1d9e3ed57207 P 697d0c986c874a3198deef9572543c12: Deleting consensus metadata
I20260620 18:43:02.740424 9742 ts_tablet_manager.cc:1905] T 7749b1ff1584458a84fc1d9e3ed57207 P 8bbe232832d34c0fb30c686d25b40730: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:43:02.741786 9237 catalog_manager.cc:4841] TS 697d0c986c874a3198deef9572543c12 (127.23.115.2:33405): tablet 7749b1ff1584458a84fc1d9e3ed57207 (table test-table [id=d240c0d53fd144e39b26ae41106b7a24]) successfully deleted
I20260620 18:43:02.743534 9743 ts_tablet_manager.cc:1905] T 7749b1ff1584458a84fc1d9e3ed57207 P 18fda0aa5eb24b54b3ce44e3e7031e33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:43:02.745796 9715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:02.719894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60976 (local address 127.23.115.1:38291)
0620 18:43:02.720093 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:02.720099 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:02.737409 (+ 17310us) server_negotiation.cc:407] Connection header received
0620 18:43:02.737583 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:02.737589 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:02.737697 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:02.737811 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:02.738469 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:02.739606 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:02.741785 (+ 2179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:02.742444 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:02.742541 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:02.744100 (+ 1559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:02.744110 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:02.744125 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:43:02.744844 (+ 719us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:02.744903 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:43:02.744927 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:02.745013 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:02.745240 (+ 227us) server_negotiation.cc:299] Negotiation successful
0620 18:43:02.745357 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 18:43:02.747155 9434 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:43:02.750530 9743 ts_tablet_manager.cc:1918] T 7749b1ff1584458a84fc1d9e3ed57207 P 18fda0aa5eb24b54b3ce44e3e7031e33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:43:02.750694 9743 log.cc:1196] T 7749b1ff1584458a84fc1d9e3ed57207 P 18fda0aa5eb24b54b3ce44e3e7031e33: Deleting WAL directory at /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980676253068-24012-0/minicluster-data/ts-2/wal/wals/7749b1ff1584458a84fc1d9e3ed57207
I20260620 18:43:02.751295 9743 ts_tablet_manager.cc:1939] T 7749b1ff1584458a84fc1d9e3ed57207 P 18fda0aa5eb24b54b3ce44e3e7031e33: Deleting consensus metadata
I20260620 18:43:02.753947 9239 catalog_manager.cc:4841] TS 18fda0aa5eb24b54b3ce44e3e7031e33 (127.23.115.3:40051): tablet 7749b1ff1584458a84fc1d9e3ed57207 (table test-table [id=d240c0d53fd144e39b26ae41106b7a24]) successfully deleted
I20260620 18:43:02.866657 9742 ts_tablet_manager.cc:1918] T 7749b1ff1584458a84fc1d9e3ed57207 P 8bbe232832d34c0fb30c686d25b40730: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:43:02.866969 9742 log.cc:1196] T 7749b1ff1584458a84fc1d9e3ed57207 P 8bbe232832d34c0fb30c686d25b40730: Deleting WAL directory at /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980676253068-24012-0/minicluster-data/ts-0/wal/wals/7749b1ff1584458a84fc1d9e3ed57207
I20260620 18:43:02.867477 9742 ts_tablet_manager.cc:1939] T 7749b1ff1584458a84fc1d9e3ed57207 P 8bbe232832d34c0fb30c686d25b40730: Deleting consensus metadata
I20260620 18:43:02.869342 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 9312
I20260620 18:43:02.869343 9238 catalog_manager.cc:4841] TS 8bbe232832d34c0fb30c686d25b40730 (127.23.115.1:38291): tablet 7749b1ff1584458a84fc1d9e3ed57207 (table test-table [id=d240c0d53fd144e39b26ae41106b7a24]) successfully deleted
I20260620 18:43:02.884210 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 9443
I20260620 18:43:02.890919 9714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:02.885162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32790 (local address 127.23.115.2:33405)
0620 18:43:02.885354 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:02.885359 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:02.885375 (+ 16us) server_negotiation.cc:407] Connection header received
0620 18:43:02.885426 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:02.885429 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:02.885516 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:02.885606 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:02.886043 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:02.886825 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:02.888063 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:02.888479 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:02.888587 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:02.889605 (+ 1018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:02.889614 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:02.889629 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:43:02.890211 (+ 582us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:02.890255 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:43:02.890274 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:02.890378 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:02.890500 (+ 122us) server_negotiation.cc:299] Negotiation successful
0620 18:43:02.890594 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 18:43:02.891868 9565 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:43:03.006312 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 9443
I20260620 18:43:03.025622 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 9573
I20260620 18:43:03.034482 9717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:03.026579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45382 (local address 127.23.115.3:40051)
0620 18:43:03.026772 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:03.026777 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:03.026865 (+ 88us) server_negotiation.cc:407] Connection header received
0620 18:43:03.027317 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:03.027322 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:03.027413 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:03.027537 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:03.028039 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:03.028796 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:03.030249 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:03.030649 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:03.030751 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:03.032484 (+ 1733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:03.032498 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:03.032518 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:43:03.033517 (+ 999us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:03.033591 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:43:03.033622 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:03.033769 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:03.034001 (+ 232us) server_negotiation.cc:299] Negotiation successful
0620 18:43:03.034153 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 18:43:03.035532 9695 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:43:03.126384 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 9573
I20260620 18:43:03.140252 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 9222
I20260620 18:43:03.141487 9283 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:43:03.258872 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 9222
2026-06-20T18:43:03Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2857 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980676253068-24012-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:43:03 dist-test-slave-d645 krb5kdc[9751](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:43:03 dist-test-slave-d645 krb5kdc[9751](info): set up 2 sockets
Jun 20 18:43:03 dist-test-slave-d645 krb5kdc[9751](info): commencing operation
krb5kdc: starting...
W20260620 18:43:04.295002 24012 mini_kdc.cc:121] Time spent starting KDC: real 0.997s user 0.005s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980676253068-24012-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980676253068-24012-0/krb5kdc/test-user.keytab.
Jun 20 18:43:04 dist-test-slave-d645 krb5kdc[9751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980984, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:43:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:43:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.115.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.115.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.115.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab.
Entry for principal kudu/127.23.115.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab.
I20260620 18:43:04.534487 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980676253068-24012-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980676253068-24012-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980676253068-24012-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.115.62:41463
--webserver_interface=127.23.115.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab
--principal=kudu/127.23.115.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:45341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.115.62:41463
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:43:04.856379 9764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:43:04.856691 9764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:43:04.856841 9764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:43:04.864451 9764 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:43:04.864569 9764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:43:04.864619 9764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:43:04.864681 9764 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:43:04.864750 9764 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:43:04.874687 9764 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:45341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980676253068-24012-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980676253068-24012-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.115.62:41463
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab
--principal=kudu/127.23.115.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.115.62:41463
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.115.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980676253068-24012-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:43:04.875995 9764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:43:04.877250 9764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:43:04.888566 9769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:04.888726 9770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:04.888656 9772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:04.889369 9764 server_base.cc:1034] running on GCE node
I20260620 18:43:04.890105 9764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:43:04.891546 9764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:43:04.892771 9764 hybrid_clock.cc:648] HybridClock initialized: now 1781980984892717 us; error 37 us; skew 500 ppm
I20260620 18:43:04.893151 9764 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:43:04 dist-test-slave-d645 krb5kdc[9751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980984, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:43:04.896248 9764 init.cc:377] Logged in from keytab as kudu/127.23.115.62@KRBTEST.COM (short username kudu)
I20260620 18:43:04.897727 9764 webserver.cc:460] Webserver started at http://127.23.115.62:36667/ using document root <none> and password file <none>
I20260620 18:43:04.898376 9764 fs_manager.cc:359] Metadata directory not provided
I20260620 18:43:04.898487 9764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:43:04.898749 9764 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:43:04.901813 9764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980676253068-24012-0/minicluster-data/master-0/data/instance:
uuid: "6308431fe2bb43b6b7fec63372e41a9c"
format_stamp: "Formatted at 2026-06-20 18:43:04 on dist-test-slave-d645"
server_key: "06b6b13a95e6e6e4225cc8d1a57cddf2"
server_key_iv: "a312722284348abfb69870f89d207b02"
server_key_version: "encryptionkey@0"
I20260620 18:43:04.902580 9764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980676253068-24012-0/minicluster-data/master-0/wal/instance:
uuid: "6308431fe2bb43b6b7fec63372e41a9c"
format_stamp: "Formatted at 2026-06-20 18:43:04 on dist-test-slave-d645"
server_key: "06b6b13a95e6e6e4225cc8d1a57cddf2"
server_key_iv: "a312722284348abfb69870f89d207b02"
server_key_version: "encryptionkey@0"
I20260620 18:43:04.907955 9764 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260620 18:43:04.911386 9779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:04.912742 9764 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 18:43:04.912900 9764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980676253068-24012-0/minicluster-data/master-0/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980676253068-24012-0/minicluster-data/master-0/wal
uuid: "6308431fe2bb43b6b7fec63372e41a9c"
format_stamp: "Formatted at 2026-06-20 18:43:04 on dist-test-slave-d645"
server_key: "06b6b13a95e6e6e4225cc8d1a57cddf2"
server_key_iv: "a312722284348abfb69870f89d207b02"
server_key_version: "encryptionkey@0"
I20260620 18:43:04.913095 9764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980676253068-24012-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980676253068-24012-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980676253068-24012-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:43:04.929817 9764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:43:04.934371 9764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:43:04.934666 9764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:43:04.953235 9764 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.62:41463
I20260620 18:43:04.953281 9830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.62:41463 every 8 connection(s)
I20260620 18:43:04.954555 9764 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
I20260620 18:43:04.957211 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 9764
I20260620 18:43:04.957404 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980676253068-24012-0/minicluster-data/master-0/wal/instance
I20260620 18:43:04.957954 24012 external_mini_cluster.cc:1428] Setting key 2c9c9b10bfccccce0876e2fb8f56f7d8
I20260620 18:43:04.958694 9831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 18:43:04 dist-test-slave-d645 krb5kdc[9751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980984, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.115.62@KRBTEST.COM
I20260620 18:43:04.973160 9831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6308431fe2bb43b6b7fec63372e41a9c: Bootstrap starting.
I20260620 18:43:04.976486 9831 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6308431fe2bb43b6b7fec63372e41a9c: Neither blocks nor log segments found. Creating new log.
I20260620 18:43:04.977559 9831 log.cc:824] T 00000000000000000000000000000000 P 6308431fe2bb43b6b7fec63372e41a9c: Log is configured to *not* fsync() on all Append() calls
I20260620 18:43:04.980944 9831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6308431fe2bb43b6b7fec63372e41a9c: No bootstrap required, opened a new log
I20260620 18:43:04.982610 9834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:04.959929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45210 (local address 127.23.115.62:41463)
0620 18:43:04.960742 (+ 813us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:04.960753 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:04.960797 (+ 44us) server_negotiation.cc:407] Connection header received
0620 18:43:04.962591 (+ 1794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:04.962613 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:04.963061 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:04.963393 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:04.965225 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:04.966551 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:04.968302 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:04.968974 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:04.969143 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:04.973898 (+ 4755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:04.973906 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:04.973921 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:04.973960 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:04.977885 (+ 3925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:04.978809 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:04.978819 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:04.978829 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:04.979129 (+ 300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:04.979613 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:04.979617 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:04.979620 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:04.980184 (+ 564us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:04.980469 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:04.981224 (+ 755us) server_negotiation.cc:299] Negotiation successful
0620 18:43:04.981635 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":372,"threads_started":1}
I20260620 18:43:04.988656 9831 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6308431fe2bb43b6b7fec63372e41a9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6308431fe2bb43b6b7fec63372e41a9c" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 41463 } }
I20260620 18:43:04.989015 9831 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6308431fe2bb43b6b7fec63372e41a9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:43:04.989110 9831 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6308431fe2bb43b6b7fec63372e41a9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6308431fe2bb43b6b7fec63372e41a9c, State: Initialized, Role: FOLLOWER
I20260620 18:43:04.989634 9831 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6308431fe2bb43b6b7fec63372e41a9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6308431fe2bb43b6b7fec63372e41a9c" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 41463 } }
I20260620 18:43:04.989833 9831 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6308431fe2bb43b6b7fec63372e41a9c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:43:04.989952 9831 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6308431fe2bb43b6b7fec63372e41a9c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:43:04.990110 9831 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6308431fe2bb43b6b7fec63372e41a9c [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:43:04.992452 9831 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6308431fe2bb43b6b7fec63372e41a9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6308431fe2bb43b6b7fec63372e41a9c" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 41463 } }
I20260620 18:43:04.992913 9831 leader_election.cc:302] T 00000000000000000000000000000000 P 6308431fe2bb43b6b7fec63372e41a9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6308431fe2bb43b6b7fec63372e41a9c; no voters:
I20260620 18:43:04.993481 9831 leader_election.cc:288] T 00000000000000000000000000000000 P 6308431fe2bb43b6b7fec63372e41a9c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:43:04.993613 9836 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6308431fe2bb43b6b7fec63372e41a9c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:43:04.994225 9836 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6308431fe2bb43b6b7fec63372e41a9c [term 1 LEADER]: Becoming Leader. State: Replica: 6308431fe2bb43b6b7fec63372e41a9c, State: Running, Role: LEADER
I20260620 18:43:04.994714 9836 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6308431fe2bb43b6b7fec63372e41a9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6308431fe2bb43b6b7fec63372e41a9c" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 41463 } }
I20260620 18:43:04.995549 9831 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6308431fe2bb43b6b7fec63372e41a9c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:43:04.997594 9838 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6308431fe2bb43b6b7fec63372e41a9c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6308431fe2bb43b6b7fec63372e41a9c. Latest consensus state: current_term: 1 leader_uuid: "6308431fe2bb43b6b7fec63372e41a9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6308431fe2bb43b6b7fec63372e41a9c" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 41463 } } }
I20260620 18:43:04.998344 9838 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6308431fe2bb43b6b7fec63372e41a9c [sys.catalog]: This master's current role is: LEADER
I20260620 18:43:04.997843 9837 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6308431fe2bb43b6b7fec63372e41a9c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6308431fe2bb43b6b7fec63372e41a9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6308431fe2bb43b6b7fec63372e41a9c" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 41463 } } }
I20260620 18:43:04.998651 9837 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6308431fe2bb43b6b7fec63372e41a9c [sys.catalog]: This master's current role is: LEADER
W20260620 18:43:05.007503 9852 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6308431fe2bb43b6b7fec63372e41a9c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 18:43:05.007673 9852 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 18:43:05.008350 9848 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:43:05.011026 9848 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:43:05.021608 9848 catalog_manager.cc:1347] Generated new cluster ID: 581d495c794f4fa7af7575de7cc49d49
I20260620 18:43:05.021782 9848 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:43:05.032308 9848 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:43:05.033627 9848 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:43:05.041239 9848 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6308431fe2bb43b6b7fec63372e41a9c: Generated new TSK 0
I20260620 18:43:05.042105 9848 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.115.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.115.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.115.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.1.keytab.
Entry for principal kudu/127.23.115.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.1.keytab.
I20260620 18:43:05.107053 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980676253068-24012-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980676253068-24012-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980676253068-24012-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980676253068-24012-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.115.1:0
--local_ip_for_outbound_sockets=127.23.115.1
--webserver_interface=127.23.115.1
--webserver_port=0
--tserver_master_addrs=127.23.115.62:41463
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.1.keytab
--principal=kudu/127.23.115.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:45341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:43:05.431072 9857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:43:05.431450 9857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:43:05.431605 9857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:43:05.440009 9857 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:43:05.440140 9857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:43:05.440305 9857 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.115.1
I20260620 18:43:05.450089 9857 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:45341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980676253068-24012-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980676253068-24012-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.1.keytab
--principal=kudu/127.23.115.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.115.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980676253068-24012-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.115.1
--webserver_port=0
--tserver_master_addrs=127.23.115.62:41463
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.115.1
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980676253068-24012-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:43:05.451382 9857 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:43:05.452602 9857 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:43:05.465310 9862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:05.465379 9863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:05.465379 9865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:05.466066 9857 server_base.cc:1034] running on GCE node
I20260620 18:43:05.466645 9857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:43:05.467608 9857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:43:05.468816 9857 hybrid_clock.cc:648] HybridClock initialized: now 1781980985468757 us; error 42 us; skew 500 ppm
I20260620 18:43:05.469138 9857 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:43:05 dist-test-slave-d645 krb5kdc[9751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980985, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:43:05.472275 9857 init.cc:377] Logged in from keytab as kudu/127.23.115.1@KRBTEST.COM (short username kudu)
I20260620 18:43:05.473650 9857 webserver.cc:460] Webserver started at http://127.23.115.1:35757/ using document root <none> and password file <none>
I20260620 18:43:05.474294 9857 fs_manager.cc:359] Metadata directory not provided
I20260620 18:43:05.474408 9857 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:43:05.474709 9857 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:43:05.477531 9857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980676253068-24012-0/minicluster-data/ts-0/data/instance:
uuid: "a6126f70151449e798ad1e66f88c6efd"
format_stamp: "Formatted at 2026-06-20 18:43:05 on dist-test-slave-d645"
server_key: "fdd9b093d8a3bdf128058722f2f65737"
server_key_iv: "d348a73b4635e53bd3f7b3b1968916c8"
server_key_version: "encryptionkey@0"
I20260620 18:43:05.478252 9857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980676253068-24012-0/minicluster-data/ts-0/wal/instance:
uuid: "a6126f70151449e798ad1e66f88c6efd"
format_stamp: "Formatted at 2026-06-20 18:43:05 on dist-test-slave-d645"
server_key: "fdd9b093d8a3bdf128058722f2f65737"
server_key_iv: "d348a73b4635e53bd3f7b3b1968916c8"
server_key_version: "encryptionkey@0"
I20260620 18:43:05.483263 9857 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260620 18:43:05.486781 9872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:05.488091 9857 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:43:05.488262 9857 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980676253068-24012-0/minicluster-data/ts-0/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980676253068-24012-0/minicluster-data/ts-0/wal
uuid: "a6126f70151449e798ad1e66f88c6efd"
format_stamp: "Formatted at 2026-06-20 18:43:05 on dist-test-slave-d645"
server_key: "fdd9b093d8a3bdf128058722f2f65737"
server_key_iv: "d348a73b4635e53bd3f7b3b1968916c8"
server_key_version: "encryptionkey@0"
I20260620 18:43:05.488445 9857 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980676253068-24012-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980676253068-24012-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980676253068-24012-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:43:05.518514 9857 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:43:05.522833 9857 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:43:05.523154 9857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:43:05.524130 9857 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:43:05.525452 9857 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:43:05.525530 9857 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:05.525655 9857 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:43:05.525708 9857 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:05.556692 9857 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.1:36321
I20260620 18:43:05.556742 9984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.1:36321 every 8 connection(s)
I20260620 18:43:05.557945 9857 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980676253068-24012-0/minicluster-data/ts-0/data/info.pb
I20260620 18:43:05.560563 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 9857
I20260620 18:43:05.560753 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980676253068-24012-0/minicluster-data/ts-0/wal/instance
I20260620 18:43:05.561201 24012 external_mini_cluster.cc:1428] Setting key d7f39ab9f28997db022fad08d8dc7d1d
Jun 20 18:43:05 dist-test-slave-d645 krb5kdc[9751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980985, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.1@KRBTEST.COM for kudu/127.23.115.62@KRBTEST.COM
I20260620 18:43:05.577431 9987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:05.560349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.1:51877 (local address 127.23.115.62:41463)
0620 18:43:05.560861 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:05.560867 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:05.561796 (+ 929us) server_negotiation.cc:407] Connection header received
0620 18:43:05.563919 (+ 2123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:05.563925 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:05.564006 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:05.564097 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:05.565587 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:05.566317 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:05.567879 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:05.568265 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:05.568370 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:05.571925 (+ 3555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:05.571931 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:05.571934 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:05.571965 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:05.574363 (+ 2398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:05.575097 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:05.575101 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:05.575103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:05.575224 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:05.575616 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:05.575620 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:05.575622 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:05.575892 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:05.576017 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:05.576733 (+ 716us) server_negotiation.cc:299] Negotiation successful
0620 18:43:05.577023 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":226,"threads_started":1}
I20260620 18:43:05.578390 9988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:05.560890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:41463 (local address 127.23.115.1:51877)
0620 18:43:05.561628 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:05.561671 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:05.563564 (+ 1893us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:05.564351 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:05.564359 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:05.564841 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:43:05.565422 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:05.565440 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:05.566470 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:05.566476 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:05.567739 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:05.567751 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:05.568477 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:05.568484 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:05.568710 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:05.569419 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:43:05.569447 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:43:05.571720 (+ 2273us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:43:05.574493 (+ 2773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:05.574501 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:05.574517 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:05.574982 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:05.575323 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:05.575328 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:05.575332 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:05.575509 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:05.576016 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:43:05.576025 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:43:05.576366 (+ 341us) client_negotiation.cc:769] Sending connection context
0620 18:43:05.576597 (+ 231us) client_negotiation.cc:240] Negotiation successful
0620 18:43:05.577209 (+ 612us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":301,"threads_started":1}
I20260620 18:43:05.580097 9985 heartbeater.cc:347] Connected to a master server at 127.23.115.62:41463
I20260620 18:43:05.580400 9985 heartbeater.cc:464] Registering TS with master...
I20260620 18:43:05.581089 9985 heartbeater.cc:511] Master 127.23.115.62:41463 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.23.115.2@KRBTEST.COM; defaulting to no policy
I20260620 18:43:05.583068 9796 ts_manager.cc:194] Registered new tserver with Master: a6126f70151449e798ad1e66f88c6efd (127.23.115.1:36321)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.115.2@KRBTEST.COM" created.
I20260620 18:43:05.584582 9796 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.115.1@KRBTEST.COM'} at 127.23.115.1:51877
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.115.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.2.keytab.
Entry for principal kudu/127.23.115.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.2.keytab.
I20260620 18:43:05.614003 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980676253068-24012-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980676253068-24012-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980676253068-24012-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980676253068-24012-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.115.2:0
--local_ip_for_outbound_sockets=127.23.115.2
--webserver_interface=127.23.115.2
--webserver_port=0
--tserver_master_addrs=127.23.115.62:41463
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.2.keytab
--principal=kudu/127.23.115.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:45341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:43:05.937556 9991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:43:05.937834 9991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:43:05.937951 9991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:43:05.945259 9991 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:43:05.945375 9991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:43:05.945514 9991 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.115.2
I20260620 18:43:05.957114 9991 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:45341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980676253068-24012-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980676253068-24012-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.2.keytab
--principal=kudu/127.23.115.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.115.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980676253068-24012-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.115.2
--webserver_port=0
--tserver_master_addrs=127.23.115.62:41463
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.115.2
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980676253068-24012-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:43:05.958540 9991 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:43:05.960192 9991 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:43:05.976399 9996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:05.976491 9997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:05.976461 9999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:05.977393 9991 server_base.cc:1034] running on GCE node
I20260620 18:43:05.977962 9991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:43:05.978853 9991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:43:05.980069 9991 hybrid_clock.cc:648] HybridClock initialized: now 1781980985980008 us; error 52 us; skew 500 ppm
I20260620 18:43:05.980361 9991 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:43:05 dist-test-slave-d645 krb5kdc[9751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980985, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:43:05.989414 9991 init.cc:377] Logged in from keytab as kudu/127.23.115.2@KRBTEST.COM (short username kudu)
I20260620 18:43:05.991000 9991 webserver.cc:460] Webserver started at http://127.23.115.2:36659/ using document root <none> and password file <none>
I20260620 18:43:05.991739 9991 fs_manager.cc:359] Metadata directory not provided
I20260620 18:43:05.991874 9991 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:43:05.992179 9991 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:43:05.995304 9991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980676253068-24012-0/minicluster-data/ts-1/data/instance:
uuid: "04390245bb3c49e8952a231e960f7ee5"
format_stamp: "Formatted at 2026-06-20 18:43:05 on dist-test-slave-d645"
server_key: "af195025e45974c1d975c506e491c3f7"
server_key_iv: "2800fc378e4c61a1c1bf726e534d2ae8"
server_key_version: "encryptionkey@0"
I20260620 18:43:05.996134 9991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980676253068-24012-0/minicluster-data/ts-1/wal/instance:
uuid: "04390245bb3c49e8952a231e960f7ee5"
format_stamp: "Formatted at 2026-06-20 18:43:05 on dist-test-slave-d645"
server_key: "af195025e45974c1d975c506e491c3f7"
server_key_iv: "2800fc378e4c61a1c1bf726e534d2ae8"
server_key_version: "encryptionkey@0"
I20260620 18:43:06.001700 9991 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:43:06.005225 10006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:06.007450 9991 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 18:43:06.007712 9991 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980676253068-24012-0/minicluster-data/ts-1/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980676253068-24012-0/minicluster-data/ts-1/wal
uuid: "04390245bb3c49e8952a231e960f7ee5"
format_stamp: "Formatted at 2026-06-20 18:43:05 on dist-test-slave-d645"
server_key: "af195025e45974c1d975c506e491c3f7"
server_key_iv: "2800fc378e4c61a1c1bf726e534d2ae8"
server_key_version: "encryptionkey@0"
I20260620 18:43:06.008143 9991 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980676253068-24012-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980676253068-24012-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980676253068-24012-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:43:06.043491 9991 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:43:06.047592 9991 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:43:06.047852 9991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:43:06.048797 9991 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:43:06.050084 9991 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:43:06.050159 9991 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:06.050282 9991 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:43:06.050333 9991 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:06.081969 9991 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.2:37151
I20260620 18:43:06.082034 10118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.2:37151 every 8 connection(s)
I20260620 18:43:06.083246 9991 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980676253068-24012-0/minicluster-data/ts-1/data/info.pb
I20260620 18:43:06.089829 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 9991
I20260620 18:43:06.090027 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980676253068-24012-0/minicluster-data/ts-1/wal/instance
I20260620 18:43:06.090569 24012 external_mini_cluster.cc:1428] Setting key 85337a0fce735eebf35fef2ccebbe9dd
Jun 20 18:43:06 dist-test-slave-d645 krb5kdc[9751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980985, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.2@KRBTEST.COM for kudu/127.23.115.62@KRBTEST.COM
I20260620 18:43:06.104522 10121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:06.085984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.2:40023 (local address 127.23.115.62:41463)
0620 18:43:06.086427 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:06.086432 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:06.087827 (+ 1395us) server_negotiation.cc:407] Connection header received
0620 18:43:06.090492 (+ 2665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:06.090497 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:06.090574 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:06.090704 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:06.092512 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:06.093277 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:06.094768 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:06.095222 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:06.095330 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:06.098638 (+ 3308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:06.098645 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:06.098649 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:06.098687 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:06.101379 (+ 2692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:06.102072 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:06.102078 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:06.102082 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:06.102219 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:06.102641 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:06.102646 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:06.102650 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:06.103027 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:06.103189 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:06.103926 (+ 737us) server_negotiation.cc:299] Negotiation successful
0620 18:43:06.104154 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":204,"threads_started":1}
I20260620 18:43:06.105484 10122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:06.086531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:41463 (local address 127.23.115.2:40023)
0620 18:43:06.087611 (+ 1080us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:06.087680 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:06.090122 (+ 2442us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:06.091001 (+ 879us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:06.091009 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:06.091578 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:43:06.092352 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:06.092374 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:06.093420 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:06.093423 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:06.094633 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:06.094645 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:06.095439 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:06.095446 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:06.095657 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:06.096281 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:43:06.096307 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:43:06.098415 (+ 2108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:43:06.101515 (+ 3100us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:06.101525 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:06.101542 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:06.101960 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:06.102319 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:06.102324 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:06.102328 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:06.102529 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:06.103163 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:43:06.103201 (+ 38us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:43:06.103579 (+ 378us) client_negotiation.cc:769] Sending connection context
0620 18:43:06.103832 (+ 253us) client_negotiation.cc:240] Negotiation successful
0620 18:43:06.104355 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":739,"thread_start_us":482,"threads_started":1}
I20260620 18:43:06.107815 10119 heartbeater.cc:347] Connected to a master server at 127.23.115.62:41463
I20260620 18:43:06.108117 10119 heartbeater.cc:464] Registering TS with master...
I20260620 18:43:06.108901 10119 heartbeater.cc:511] Master 127.23.115.62:41463 requested a full tablet report, sending...
I20260620 18:43:06.110543 9796 ts_manager.cc:194] Registered new tserver with Master: 04390245bb3c49e8952a231e960f7ee5 (127.23.115.2:37151)
I20260620 18:43:06.111517 9796 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.115.2@KRBTEST.COM'} at 127.23.115.2:40023
WARNING: no policy specified for kudu/127.23.115.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.115.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.115.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.3.keytab.
Entry for principal kudu/127.23.115.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.3.keytab.
I20260620 18:43:06.144306 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980676253068-24012-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980676253068-24012-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980676253068-24012-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980676253068-24012-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.115.3:0
--local_ip_for_outbound_sockets=127.23.115.3
--webserver_interface=127.23.115.3
--webserver_port=0
--tserver_master_addrs=127.23.115.62:41463
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.3.keytab
--principal=kudu/127.23.115.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:45341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:43:06.468341 10125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:43:06.468619 10125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:43:06.468714 10125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:43:06.476115 10125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:43:06.476228 10125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:43:06.476348 10125 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.115.3
I20260620 18:43:06.488334 10125 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:45341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980676253068-24012-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980676253068-24012-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.3.keytab
--principal=kudu/127.23.115.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.115.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980676253068-24012-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.115.3
--webserver_port=0
--tserver_master_addrs=127.23.115.62:41463
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.115.3
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980676253068-24012-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:43:06.489598 10125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:43:06.490805 10125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:43:06.504647 10133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:06.504612 10131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:06.504817 10130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:06.505697 10125 server_base.cc:1034] running on GCE node
I20260620 18:43:06.506357 10125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:43:06.507481 10125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:43:06.508708 10125 hybrid_clock.cc:648] HybridClock initialized: now 1781980986508646 us; error 50 us; skew 500 ppm
I20260620 18:43:06.509073 10125 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:43:06 dist-test-slave-d645 krb5kdc[9751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980986, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:43:06.512217 10125 init.cc:377] Logged in from keytab as kudu/127.23.115.3@KRBTEST.COM (short username kudu)
I20260620 18:43:06.513697 10125 webserver.cc:460] Webserver started at http://127.23.115.3:35319/ using document root <none> and password file <none>
I20260620 18:43:06.514360 10125 fs_manager.cc:359] Metadata directory not provided
I20260620 18:43:06.514460 10125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:43:06.514707 10125 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:43:06.517716 10125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980676253068-24012-0/minicluster-data/ts-2/data/instance:
uuid: "293e2d88c09146839377760014804070"
format_stamp: "Formatted at 2026-06-20 18:43:06 on dist-test-slave-d645"
server_key: "d6569c7f1541959e2ed4127307cadad8"
server_key_iv: "ec4e22da9da27ca70a160164b73e2dc1"
server_key_version: "encryptionkey@0"
I20260620 18:43:06.518445 10125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980676253068-24012-0/minicluster-data/ts-2/wal/instance:
uuid: "293e2d88c09146839377760014804070"
format_stamp: "Formatted at 2026-06-20 18:43:06 on dist-test-slave-d645"
server_key: "d6569c7f1541959e2ed4127307cadad8"
server_key_iv: "ec4e22da9da27ca70a160164b73e2dc1"
server_key_version: "encryptionkey@0"
I20260620 18:43:06.523490 10125 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260620 18:43:06.527010 10140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:06.528360 10125 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:43:06.528518 10125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980676253068-24012-0/minicluster-data/ts-2/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980676253068-24012-0/minicluster-data/ts-2/wal
uuid: "293e2d88c09146839377760014804070"
format_stamp: "Formatted at 2026-06-20 18:43:06 on dist-test-slave-d645"
server_key: "d6569c7f1541959e2ed4127307cadad8"
server_key_iv: "ec4e22da9da27ca70a160164b73e2dc1"
server_key_version: "encryptionkey@0"
I20260620 18:43:06.528699 10125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980676253068-24012-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980676253068-24012-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980676253068-24012-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:43:06.547838 10125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:43:06.551893 10125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:43:06.552161 10125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:43:06.553122 10125 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:43:06.554442 10125 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:43:06.554517 10125 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:06.554626 10125 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:43:06.554677 10125 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:06.586483 10125 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.3:45923
I20260620 18:43:06.586548 10252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.3:45923 every 8 connection(s)
I20260620 18:43:06.587807 10125 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980676253068-24012-0/minicluster-data/ts-2/data/info.pb
I20260620 18:43:06.587863 9985 heartbeater.cc:503] Master 127.23.115.62:41463 was elected leader, sending a full tablet report...
I20260620 18:43:06.597949 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 10125
I20260620 18:43:06.598135 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980676253068-24012-0/minicluster-data/ts-2/wal/instance
I20260620 18:43:06.598825 24012 external_mini_cluster.cc:1428] Setting key fc7cb6553f6bbfb404fe38592de0f0f2
Jun 20 18:43:06 dist-test-slave-d645 krb5kdc[9751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980986, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.3@KRBTEST.COM for kudu/127.23.115.62@KRBTEST.COM
I20260620 18:43:06.609418 10121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:06.590225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.3:39463 (local address 127.23.115.62:41463)
0620 18:43:06.590393 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:06.590398 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:06.591649 (+ 1251us) server_negotiation.cc:407] Connection header received
0620 18:43:06.593572 (+ 1923us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:06.593577 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:06.593663 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:06.593759 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:06.595396 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:06.596215 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:06.597712 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:06.598153 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:06.598275 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:06.602712 (+ 4437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:06.602717 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:06.602720 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:06.602754 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:06.605267 (+ 2513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:06.606479 (+ 1212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:06.606484 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:06.606487 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:06.606659 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:06.607198 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:06.607202 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:06.607204 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:06.607570 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:06.607742 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:06.608616 (+ 874us) server_negotiation.cc:299] Negotiation successful
0620 18:43:06.608953 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 18:43:06.611155 10255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:06.590700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:41463 (local address 127.23.115.3:39463)
0620 18:43:06.591493 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:06.591542 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:06.593299 (+ 1757us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:06.594000 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:06.594014 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:06.594503 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:43:06.595221 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:06.595249 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:06.596356 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:06.596362 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:06.597569 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:06.597582 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:06.598382 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:06.598389 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:06.598604 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:06.599636 (+ 1032us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:43:06.599665 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:43:06.602535 (+ 2870us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:43:06.605519 (+ 2984us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:06.605535 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:06.605555 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:06.606215 (+ 660us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:06.606787 (+ 572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:06.606793 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:06.606795 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:06.607076 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:06.607708 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:43:06.607719 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:43:06.608187 (+ 468us) client_negotiation.cc:769] Sending connection context
0620 18:43:06.608494 (+ 307us) client_negotiation.cc:240] Negotiation successful
0620 18:43:06.609216 (+ 722us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":307,"threads_started":1}
I20260620 18:43:06.613283 10253 heartbeater.cc:347] Connected to a master server at 127.23.115.62:41463
I20260620 18:43:06.613682 10253 heartbeater.cc:464] Registering TS with master...
I20260620 18:43:06.614655 10253 heartbeater.cc:511] Master 127.23.115.62:41463 requested a full tablet report, sending...
I20260620 18:43:06.616259 9796 ts_manager.cc:194] Registered new tserver with Master: 293e2d88c09146839377760014804070 (127.23.115.3:45923)
I20260620 18:43:06.617091 9796 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.115.3@KRBTEST.COM'} at 127.23.115.3:39463
I20260620 18:43:06.628146 24012 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:43:06.647864 10121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:06.633262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45220 (local address 127.23.115.62:41463)
0620 18:43:06.633481 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:06.633489 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:06.634000 (+ 511us) server_negotiation.cc:407] Connection header received
0620 18:43:06.634116 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:06.634120 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:06.634237 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:06.634368 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:06.636030 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:06.636805 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:06.638641 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:06.639210 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:06.639345 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:06.642086 (+ 2741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:06.642092 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:06.642095 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:06.642129 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:06.644601 (+ 2472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:06.645432 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:06.645436 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:06.645439 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:06.645589 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:06.646078 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:06.646082 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:06.646084 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:06.646392 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:06.646607 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:06.647229 (+ 622us) server_negotiation.cc:299] Negotiation successful
0620 18:43:06.647556 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 18:43:06.652446 9796 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45220:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:43:06.659157 9796 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:43:06.684360 10265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:06.672424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.3:45923 (local address 127.0.0.1:53824)
0620 18:43:06.673143 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:06.673187 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:06.673379 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:06.675741 (+ 2362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:06.675755 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:06.675793 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:43:06.676289 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:06.676306 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:06.678398 (+ 2092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:06.678404 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:06.680847 (+ 2443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:06.680863 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:06.683451 (+ 2588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:06.683457 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:06.683666 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:06.683726 (+ 60us) client_negotiation.cc:769] Sending connection context
0620 18:43:06.683835 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 18:43:06.684023 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"mutex_wait_us":1,"thread_start_us":298,"threads_started":1}
I20260620 18:43:06.685346 10266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:06.672687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53824 (local address 127.23.115.3:45923)
0620 18:43:06.674346 (+ 1659us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:06.674356 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:06.674391 (+ 35us) server_negotiation.cc:407] Connection header received
0620 18:43:06.674496 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:06.674504 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:06.674763 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:06.675096 (+ 333us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:43:06.676505 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:06.678219 (+ 1714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:06.681096 (+ 2877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:06.682679 (+ 1583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:06.684612 (+ 1933us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:06.684855 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:06.684938 (+ 83us) server_negotiation.cc:299] Negotiation successful
0620 18:43:06.685090 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1467,"thread_start_us":1101,"threads_started":1}
I20260620 18:43:06.685947 10267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:06.672804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47614 (local address 127.23.115.1:36321)
0620 18:43:06.675830 (+ 3026us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:06.675838 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:06.676436 (+ 598us) server_negotiation.cc:407] Connection header received
0620 18:43:06.676723 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:06.676731 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:06.676990 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:06.677146 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:43:06.678086 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:06.679797 (+ 1711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:06.683090 (+ 3293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:06.684505 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:06.684624 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:06.684845 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:06.685574 (+ 729us) server_negotiation.cc:299] Negotiation successful
0620 18:43:06.685705 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2822,"thread_start_us":2675,"threads_started":1}
I20260620 18:43:06.686668 10270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:06.672425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.1:36321 (local address 127.0.0.1:47614)
0620 18:43:06.676322 (+ 3897us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:06.676344 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:06.676547 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:06.677414 (+ 867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:06.677420 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:06.677447 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:43:06.677890 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:06.677901 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:06.679997 (+ 2096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:06.680004 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:06.682391 (+ 2387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:06.682408 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:06.685027 (+ 2619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:06.685034 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:06.685204 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:06.685238 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 18:43:06.685339 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 18:43:06.686238 (+ 899us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3758,"mutex_wait_us":531,"thread_start_us":9810,"threads_started":1}
I20260620 18:43:06.689284 10188 tablet_service.cc:1467] Processing CreateTablet for tablet 8e116c8753d441948f655c57bfdf44f7 (DEFAULT_TABLE table=test-table [id=49f46a4671674230918117429e1746b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:43:06.690076 9920 tablet_service.cc:1467] Processing CreateTablet for tablet 8e116c8753d441948f655c57bfdf44f7 (DEFAULT_TABLE table=test-table [id=49f46a4671674230918117429e1746b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:43:06.690820 10188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e116c8753d441948f655c57bfdf44f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:43:06.691310 9920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e116c8753d441948f655c57bfdf44f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:43:06.691409 10268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:06.673548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.2:37151 (local address 127.0.0.1:36694)
0620 18:43:06.683079 (+ 9531us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:06.683098 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:06.683216 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:06.684133 (+ 917us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:06.684138 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:06.684164 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:43:06.684524 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:06.684535 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:06.686349 (+ 1814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:06.686354 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:06.688501 (+ 2147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:06.688515 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:06.690620 (+ 2105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:06.690629 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:06.690807 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:06.690840 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 18:43:06.690966 (+ 126us) client_negotiation.cc:240] Negotiation successful
0620 18:43:06.691098 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9412,"thread_start_us":2468,"threads_started":1}
I20260620 18:43:06.691864 10269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:06.674073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36694 (local address 127.23.115.2:37151)
0620 18:43:06.683036 (+ 8963us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:06.683045 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:06.683192 (+ 147us) server_negotiation.cc:407] Connection header received
0620 18:43:06.683477 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:06.683484 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:06.683715 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:06.683920 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:43:06.684694 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:06.686203 (+ 1509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:06.688663 (+ 2460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:06.690321 (+ 1658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:06.690464 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:06.690646 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:06.691069 (+ 423us) server_negotiation.cc:299] Negotiation successful
0620 18:43:06.691229 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8758,"thread_start_us":9127,"threads_started":1}
I20260620 18:43:06.695374 10054 tablet_service.cc:1467] Processing CreateTablet for tablet 8e116c8753d441948f655c57bfdf44f7 (DEFAULT_TABLE table=test-table [id=49f46a4671674230918117429e1746b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:43:06.696643 10054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e116c8753d441948f655c57bfdf44f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:43:06.703061 10271 tablet_bootstrap.cc:492] T 8e116c8753d441948f655c57bfdf44f7 P a6126f70151449e798ad1e66f88c6efd: Bootstrap starting.
I20260620 18:43:06.704949 10272 tablet_bootstrap.cc:492] T 8e116c8753d441948f655c57bfdf44f7 P 293e2d88c09146839377760014804070: Bootstrap starting.
I20260620 18:43:06.708267 10271 tablet_bootstrap.cc:654] T 8e116c8753d441948f655c57bfdf44f7 P a6126f70151449e798ad1e66f88c6efd: Neither blocks nor log segments found. Creating new log.
I20260620 18:43:06.708321 10272 tablet_bootstrap.cc:654] T 8e116c8753d441948f655c57bfdf44f7 P 293e2d88c09146839377760014804070: Neither blocks nor log segments found. Creating new log.
I20260620 18:43:06.709532 10272 log.cc:824] T 8e116c8753d441948f655c57bfdf44f7 P 293e2d88c09146839377760014804070: Log is configured to *not* fsync() on all Append() calls
I20260620 18:43:06.709532 10271 log.cc:824] T 8e116c8753d441948f655c57bfdf44f7 P a6126f70151449e798ad1e66f88c6efd: Log is configured to *not* fsync() on all Append() calls
I20260620 18:43:06.716392 10272 tablet_bootstrap.cc:492] T 8e116c8753d441948f655c57bfdf44f7 P 293e2d88c09146839377760014804070: No bootstrap required, opened a new log
I20260620 18:43:06.716439 10271 tablet_bootstrap.cc:492] T 8e116c8753d441948f655c57bfdf44f7 P a6126f70151449e798ad1e66f88c6efd: No bootstrap required, opened a new log
I20260620 18:43:06.716744 10271 ts_tablet_manager.cc:1397] T 8e116c8753d441948f655c57bfdf44f7 P a6126f70151449e798ad1e66f88c6efd: Time spent bootstrapping tablet: real 0.014s user 0.010s sys 0.000s
I20260620 18:43:06.716909 10272 ts_tablet_manager.cc:1397] T 8e116c8753d441948f655c57bfdf44f7 P 293e2d88c09146839377760014804070: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.005s
I20260620 18:43:06.717371 10275 tablet_bootstrap.cc:492] T 8e116c8753d441948f655c57bfdf44f7 P 04390245bb3c49e8952a231e960f7ee5: Bootstrap starting.
I20260620 18:43:06.720983 10275 tablet_bootstrap.cc:654] T 8e116c8753d441948f655c57bfdf44f7 P 04390245bb3c49e8952a231e960f7ee5: Neither blocks nor log segments found. Creating new log.
I20260620 18:43:06.722210 10275 log.cc:824] T 8e116c8753d441948f655c57bfdf44f7 P 04390245bb3c49e8952a231e960f7ee5: Log is configured to *not* fsync() on all Append() calls
I20260620 18:43:06.724277 10271 raft_consensus.cc:348] T 8e116c8753d441948f655c57bfdf44f7 P a6126f70151449e798ad1e66f88c6efd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "293e2d88c09146839377760014804070" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 45923 } } peers { permanent_uuid: "04390245bb3c49e8952a231e960f7ee5" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 37151 } } peers { permanent_uuid: "a6126f70151449e798ad1e66f88c6efd" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 36321 } }
I20260620 18:43:06.724617 10271 raft_consensus.cc:374] T 8e116c8753d441948f655c57bfdf44f7 P a6126f70151449e798ad1e66f88c6efd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:43:06.724714 10271 raft_consensus.cc:729] T 8e116c8753d441948f655c57bfdf44f7 P a6126f70151449e798ad1e66f88c6efd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6126f70151449e798ad1e66f88c6efd, State: Initialized, Role: FOLLOWER
I20260620 18:43:06.725258 10271 consensus_queue.cc:260] T 8e116c8753d441948f655c57bfdf44f7 P a6126f70151449e798ad1e66f88c6efd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "293e2d88c09146839377760014804070" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 45923 } } peers { permanent_uuid: "04390245bb3c49e8952a231e960f7ee5" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 37151 } } peers { permanent_uuid: "a6126f70151449e798ad1e66f88c6efd" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 36321 } }
I20260620 18:43:06.725427 10275 tablet_bootstrap.cc:492] T 8e116c8753d441948f655c57bfdf44f7 P 04390245bb3c49e8952a231e960f7ee5: No bootstrap required, opened a new log
I20260620 18:43:06.725679 10275 ts_tablet_manager.cc:1397] T 8e116c8753d441948f655c57bfdf44f7 P 04390245bb3c49e8952a231e960f7ee5: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260620 18:43:06.726923 10271 ts_tablet_manager.cc:1428] T 8e116c8753d441948f655c57bfdf44f7 P a6126f70151449e798ad1e66f88c6efd: Time spent starting tablet: real 0.010s user 0.007s sys 0.004s
I20260620 18:43:06.726709 10272 raft_consensus.cc:348] T 8e116c8753d441948f655c57bfdf44f7 P 293e2d88c09146839377760014804070 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "293e2d88c09146839377760014804070" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 45923 } } peers { permanent_uuid: "04390245bb3c49e8952a231e960f7ee5" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 37151 } } peers { permanent_uuid: "a6126f70151449e798ad1e66f88c6efd" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 36321 } }
I20260620 18:43:06.727159 10272 raft_consensus.cc:374] T 8e116c8753d441948f655c57bfdf44f7 P 293e2d88c09146839377760014804070 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:43:06.727274 10272 raft_consensus.cc:729] T 8e116c8753d441948f655c57bfdf44f7 P 293e2d88c09146839377760014804070 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 293e2d88c09146839377760014804070, State: Initialized, Role: FOLLOWER
I20260620 18:43:06.727953 10272 consensus_queue.cc:260] T 8e116c8753d441948f655c57bfdf44f7 P 293e2d88c09146839377760014804070 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "293e2d88c09146839377760014804070" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 45923 } } peers { permanent_uuid: "04390245bb3c49e8952a231e960f7ee5" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 37151 } } peers { permanent_uuid: "a6126f70151449e798ad1e66f88c6efd" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 36321 } }
I20260620 18:43:06.729027 10253 heartbeater.cc:503] Master 127.23.115.62:41463 was elected leader, sending a full tablet report...
I20260620 18:43:06.729580 10272 ts_tablet_manager.cc:1428] T 8e116c8753d441948f655c57bfdf44f7 P 293e2d88c09146839377760014804070: Time spent starting tablet: real 0.013s user 0.008s sys 0.003s
I20260620 18:43:06.735332 10275 raft_consensus.cc:348] T 8e116c8753d441948f655c57bfdf44f7 P 04390245bb3c49e8952a231e960f7ee5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "293e2d88c09146839377760014804070" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 45923 } } peers { permanent_uuid: "04390245bb3c49e8952a231e960f7ee5" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 37151 } } peers { permanent_uuid: "a6126f70151449e798ad1e66f88c6efd" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 36321 } }
I20260620 18:43:06.735684 10275 raft_consensus.cc:374] T 8e116c8753d441948f655c57bfdf44f7 P 04390245bb3c49e8952a231e960f7ee5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:43:06.735773 10275 raft_consensus.cc:729] T 8e116c8753d441948f655c57bfdf44f7 P 04390245bb3c49e8952a231e960f7ee5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04390245bb3c49e8952a231e960f7ee5, State: Initialized, Role: FOLLOWER
I20260620 18:43:06.736285 10275 consensus_queue.cc:260] T 8e116c8753d441948f655c57bfdf44f7 P 04390245bb3c49e8952a231e960f7ee5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "293e2d88c09146839377760014804070" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 45923 } } peers { permanent_uuid: "04390245bb3c49e8952a231e960f7ee5" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 37151 } } peers { permanent_uuid: "a6126f70151449e798ad1e66f88c6efd" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 36321 } }
I20260620 18:43:06.737243 10119 heartbeater.cc:503] Master 127.23.115.62:41463 was elected leader, sending a full tablet report...
I20260620 18:43:06.737653 10275 ts_tablet_manager.cc:1428] T 8e116c8753d441948f655c57bfdf44f7 P 04390245bb3c49e8952a231e960f7ee5: Time spent starting tablet: real 0.012s user 0.008s sys 0.005s
I20260620 18:43:06.801432 10278 raft_consensus.cc:482] T 8e116c8753d441948f655c57bfdf44f7 P 293e2d88c09146839377760014804070 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:43:06.801718 10278 raft_consensus.cc:504] T 8e116c8753d441948f655c57bfdf44f7 P 293e2d88c09146839377760014804070 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "293e2d88c09146839377760014804070" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 45923 } } peers { permanent_uuid: "04390245bb3c49e8952a231e960f7ee5" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 37151 } } peers { permanent_uuid: "a6126f70151449e798ad1e66f88c6efd" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 36321 } }
I20260620 18:43:06.803557 10278 leader_election.cc:288] T 8e116c8753d441948f655c57bfdf44f7 P 293e2d88c09146839377760014804070 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 04390245bb3c49e8952a231e960f7ee5 (127.23.115.2:37151), a6126f70151449e798ad1e66f88c6efd (127.23.115.1:36321)
W20260620 18:43:06.810312 9986 tablet.cc:2369] T 8e116c8753d441948f655c57bfdf44f7 P a6126f70151449e798ad1e66f88c6efd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:43:06.811662 10267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:06.804577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.3:34419 (local address 127.23.115.1:36321)
0620 18:43:06.804779 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:06.804785 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:06.804801 (+ 16us) server_negotiation.cc:407] Connection header received
0620 18:43:06.804872 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:06.804877 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:06.804950 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:06.805044 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:43:06.805492 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:06.806607 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:06.809134 (+ 2527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:06.810664 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:06.811007 (+ 343us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:06.811089 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:06.811327 (+ 238us) server_negotiation.cc:299] Negotiation successful
0620 18:43:06.811437 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 18:43:06.811962 10269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:06.803900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.3:33699 (local address 127.23.115.2:37151)
0620 18:43:06.804105 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:06.804112 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:06.804762 (+ 650us) server_negotiation.cc:407] Connection header received
0620 18:43:06.804875 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:06.804880 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:06.804950 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:06.805044 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:43:06.805688 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:06.806742 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:06.809134 (+ 2392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:06.810664 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:06.811020 (+ 356us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:06.811090 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:06.811570 (+ 480us) server_negotiation.cc:299] Negotiation successful
0620 18:43:06.811712 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 18:43:06.811967 10280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:06.803888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.2:37151 (local address 127.23.115.3:33699)
0620 18:43:06.804665 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:06.804681 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:06.804776 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:06.805098 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:06.805130 (+ 32us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:06.805148 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:43:06.805555 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:06.805565 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:06.806894 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:06.806899 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:06.808970 (+ 2071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:06.808979 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:06.811254 (+ 2275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:06.811260 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:06.811418 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:06.811445 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 18:43:06.811526 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 18:43:06.811649 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":671,"mutex_wait_us":16,"spinlock_wait_cycles":21376,"thread_start_us":447,"threads_started":1}
I20260620 18:43:06.812109 10255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:06.803830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.1:36321 (local address 127.23.115.3:34419)
0620 18:43:06.804122 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:06.804138 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:06.804256 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:06.805098 (+ 842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:06.805103 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:06.805133 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:43:06.805353 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:06.805365 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:06.806779 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:06.806785 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:06.808956 (+ 2171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:06.808972 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:06.810916 (+ 1944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:06.810922 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:06.811140 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:06.811176 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 18:43:06.811288 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 18:43:06.811437 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":76}
I20260620 18:43:06.812974 10074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e116c8753d441948f655c57bfdf44f7" candidate_uuid: "293e2d88c09146839377760014804070" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04390245bb3c49e8952a231e960f7ee5" is_pre_election: true
I20260620 18:43:06.812999 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e116c8753d441948f655c57bfdf44f7" candidate_uuid: "293e2d88c09146839377760014804070" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6126f70151449e798ad1e66f88c6efd" is_pre_election: true
I20260620 18:43:06.813446 9940 raft_consensus.cc:2454] T 8e116c8753d441948f655c57bfdf44f7 P a6126f70151449e798ad1e66f88c6efd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 293e2d88c09146839377760014804070 in term 0.
I20260620 18:43:06.813716 10074 raft_consensus.cc:2454] T 8e116c8753d441948f655c57bfdf44f7 P 04390245bb3c49e8952a231e960f7ee5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 293e2d88c09146839377760014804070 in term 0.
I20260620 18:43:06.814116 10141 leader_election.cc:302] T 8e116c8753d441948f655c57bfdf44f7 P 293e2d88c09146839377760014804070 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 293e2d88c09146839377760014804070, a6126f70151449e798ad1e66f88c6efd; no voters:
I20260620 18:43:06.814582 10278 raft_consensus.cc:2781] T 8e116c8753d441948f655c57bfdf44f7 P 293e2d88c09146839377760014804070 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:43:06.814719 10278 raft_consensus.cc:482] T 8e116c8753d441948f655c57bfdf44f7 P 293e2d88c09146839377760014804070 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:43:06.814806 10278 raft_consensus.cc:3037] T 8e116c8753d441948f655c57bfdf44f7 P 293e2d88c09146839377760014804070 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:43:06.817071 10278 raft_consensus.cc:504] T 8e116c8753d441948f655c57bfdf44f7 P 293e2d88c09146839377760014804070 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "293e2d88c09146839377760014804070" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 45923 } } peers { permanent_uuid: "04390245bb3c49e8952a231e960f7ee5" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 37151 } } peers { permanent_uuid: "a6126f70151449e798ad1e66f88c6efd" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 36321 } }
I20260620 18:43:06.817647 10278 leader_election.cc:288] T 8e116c8753d441948f655c57bfdf44f7 P 293e2d88c09146839377760014804070 [CANDIDATE]: Term 1 election: Requested vote from peers 04390245bb3c49e8952a231e960f7ee5 (127.23.115.2:37151), a6126f70151449e798ad1e66f88c6efd (127.23.115.1:36321)
I20260620 18:43:06.818116 10074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e116c8753d441948f655c57bfdf44f7" candidate_uuid: "293e2d88c09146839377760014804070" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04390245bb3c49e8952a231e960f7ee5"
I20260620 18:43:06.818128 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e116c8753d441948f655c57bfdf44f7" candidate_uuid: "293e2d88c09146839377760014804070" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6126f70151449e798ad1e66f88c6efd"
I20260620 18:43:06.818305 9940 raft_consensus.cc:3037] T 8e116c8753d441948f655c57bfdf44f7 P a6126f70151449e798ad1e66f88c6efd [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:43:06.818313 10074 raft_consensus.cc:3037] T 8e116c8753d441948f655c57bfdf44f7 P 04390245bb3c49e8952a231e960f7ee5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:43:06.820472 9940 raft_consensus.cc:2454] T 8e116c8753d441948f655c57bfdf44f7 P a6126f70151449e798ad1e66f88c6efd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 293e2d88c09146839377760014804070 in term 1.
I20260620 18:43:06.820500 10074 raft_consensus.cc:2454] T 8e116c8753d441948f655c57bfdf44f7 P 04390245bb3c49e8952a231e960f7ee5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 293e2d88c09146839377760014804070 in term 1.
I20260620 18:43:06.820986 10141 leader_election.cc:302] T 8e116c8753d441948f655c57bfdf44f7 P 293e2d88c09146839377760014804070 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 04390245bb3c49e8952a231e960f7ee5, 293e2d88c09146839377760014804070, a6126f70151449e798ad1e66f88c6efd; no voters:
I20260620 18:43:06.821348 10278 raft_consensus.cc:2781] T 8e116c8753d441948f655c57bfdf44f7 P 293e2d88c09146839377760014804070 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:43:06.821765 10278 raft_consensus.cc:686] T 8e116c8753d441948f655c57bfdf44f7 P 293e2d88c09146839377760014804070 [term 1 LEADER]: Becoming Leader. State: Replica: 293e2d88c09146839377760014804070, State: Running, Role: LEADER
I20260620 18:43:06.822204 10278 consensus_queue.cc:237] T 8e116c8753d441948f655c57bfdf44f7 P 293e2d88c09146839377760014804070 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "293e2d88c09146839377760014804070" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 45923 } } peers { permanent_uuid: "04390245bb3c49e8952a231e960f7ee5" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 37151 } } peers { permanent_uuid: "a6126f70151449e798ad1e66f88c6efd" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 36321 } }
I20260620 18:43:06.826771 9796 catalog_manager.cc:5495] T 8e116c8753d441948f655c57bfdf44f7 P 293e2d88c09146839377760014804070 reported cstate change: term changed from 0 to 1, leader changed from <none> to 293e2d88c09146839377760014804070 (127.23.115.3). New cstate: current_term: 1 leader_uuid: "293e2d88c09146839377760014804070" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "293e2d88c09146839377760014804070" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 45923 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "04390245bb3c49e8952a231e960f7ee5" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 37151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6126f70151449e798ad1e66f88c6efd" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 36321 } health_report { overall_health: UNKNOWN } } }
W20260620 18:43:06.835248 10120 tablet.cc:2369] T 8e116c8753d441948f655c57bfdf44f7 P 04390245bb3c49e8952a231e960f7ee5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:43:06.840307 10254 tablet.cc:2369] T 8e116c8753d441948f655c57bfdf44f7 P 293e2d88c09146839377760014804070: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:43:06.844537 10266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:06.838789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53830 (local address 127.23.115.3:45923)
0620 18:43:06.839043 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:06.839047 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:06.839068 (+ 21us) server_negotiation.cc:407] Connection header received
0620 18:43:06.839258 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:06.839261 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:06.839352 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:06.839472 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:43:06.839867 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:06.840633 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:06.842086 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:06.842512 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:06.842611 (+ 99us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:06.843327 (+ 716us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:43:06.843828 (+ 501us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:43:06.843963 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:06.844257 (+ 294us) server_negotiation.cc:299] Negotiation successful
0620 18:43:06.844310 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 18:43:06.852272 9940 raft_consensus.cc:1261] T 8e116c8753d441948f655c57bfdf44f7 P a6126f70151449e798ad1e66f88c6efd [term 1 FOLLOWER]: Refusing update from remote peer 293e2d88c09146839377760014804070: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:43:06.852372 10074 raft_consensus.cc:1261] T 8e116c8753d441948f655c57bfdf44f7 P 04390245bb3c49e8952a231e960f7ee5 [term 1 FOLLOWER]: Refusing update from remote peer 293e2d88c09146839377760014804070: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:43:06.853233 10281 consensus_queue.cc:1035] T 8e116c8753d441948f655c57bfdf44f7 P 293e2d88c09146839377760014804070 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6126f70151449e798ad1e66f88c6efd" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 36321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:43:06.853592 10278 consensus_queue.cc:1035] T 8e116c8753d441948f655c57bfdf44f7 P 293e2d88c09146839377760014804070 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04390245bb3c49e8952a231e960f7ee5" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 37151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:43:06.864466 10284 mvcc.cc:205] Tried to move back new op lower bound from 7298994122137456640 to 7298994122027474944. Current Snapshot: MvccSnapshot[applied={T|T < 7298994122137456640}]
I20260620 18:43:06.865641 10286 mvcc.cc:205] Tried to move back new op lower bound from 7298994122137456640 to 7298994122027474944. Current Snapshot: MvccSnapshot[applied={T|T < 7298994122137456640}]
I20260620 18:43:06.878401 9796 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45220:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:43:06.878650 9796 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45220:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:43:06.882381 9796 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6308431fe2bb43b6b7fec63372e41a9c: Sending DeleteTablet for 3 replicas of tablet 8e116c8753d441948f655c57bfdf44f7
I20260620 18:43:06.883636 10054 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e116c8753d441948f655c57bfdf44f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:43:06 UTC) from {username='kudu', principal='kudu/127.23.115.62@KRBTEST.COM'} at 127.0.0.1:36694
I20260620 18:43:06.883723 10188 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e116c8753d441948f655c57bfdf44f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:43:06 UTC) from {username='kudu', principal='kudu/127.23.115.62@KRBTEST.COM'} at 127.0.0.1:53824
I20260620 18:43:06.883785 9920 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e116c8753d441948f655c57bfdf44f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:43:06 UTC) from {username='kudu', principal='kudu/127.23.115.62@KRBTEST.COM'} at 127.0.0.1:47614
I20260620 18:43:06.891268 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 9857
I20260620 18:43:06.900053 10295 tablet_replica.cc:330] T 8e116c8753d441948f655c57bfdf44f7 P 04390245bb3c49e8952a231e960f7ee5: stopping tablet replica
I20260620 18:43:06.900545 10294 tablet_replica.cc:330] T 8e116c8753d441948f655c57bfdf44f7 P 293e2d88c09146839377760014804070: stopping tablet replica
I20260620 18:43:06.900626 10295 raft_consensus.cc:2229] T 8e116c8753d441948f655c57bfdf44f7 P 04390245bb3c49e8952a231e960f7ee5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:43:06.900970 10294 raft_consensus.cc:2229] T 8e116c8753d441948f655c57bfdf44f7 P 293e2d88c09146839377760014804070 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:43:06.901151 10295 raft_consensus.cc:2258] T 8e116c8753d441948f655c57bfdf44f7 P 04390245bb3c49e8952a231e960f7ee5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:43:06.901614 10294 raft_consensus.cc:2258] T 8e116c8753d441948f655c57bfdf44f7 P 293e2d88c09146839377760014804070 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:43:06.903501 10295 ts_tablet_manager.cc:1905] T 8e116c8753d441948f655c57bfdf44f7 P 04390245bb3c49e8952a231e960f7ee5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:43:06.903514 10294 ts_tablet_manager.cc:1905] T 8e116c8753d441948f655c57bfdf44f7 P 293e2d88c09146839377760014804070: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 20 18:43:06 dist-test-slave-d645 krb5kdc[9751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980984, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.115.1@KRBTEST.COM
I20260620 18:43:06.910753 10294 ts_tablet_manager.cc:1918] T 8e116c8753d441948f655c57bfdf44f7 P 293e2d88c09146839377760014804070: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:43:06.910777 10295 ts_tablet_manager.cc:1918] T 8e116c8753d441948f655c57bfdf44f7 P 04390245bb3c49e8952a231e960f7ee5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:43:06.910950 10294 log.cc:1196] T 8e116c8753d441948f655c57bfdf44f7 P 293e2d88c09146839377760014804070: Deleting WAL directory at /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980676253068-24012-0/minicluster-data/ts-2/wal/wals/8e116c8753d441948f655c57bfdf44f7
I20260620 18:43:06.910979 10295 log.cc:1196] T 8e116c8753d441948f655c57bfdf44f7 P 04390245bb3c49e8952a231e960f7ee5: Deleting WAL directory at /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980676253068-24012-0/minicluster-data/ts-1/wal/wals/8e116c8753d441948f655c57bfdf44f7
I20260620 18:43:06.911550 10294 ts_tablet_manager.cc:1939] T 8e116c8753d441948f655c57bfdf44f7 P 293e2d88c09146839377760014804070: Deleting consensus metadata
I20260620 18:43:06.911851 10295 ts_tablet_manager.cc:1939] T 8e116c8753d441948f655c57bfdf44f7 P 04390245bb3c49e8952a231e960f7ee5: Deleting consensus metadata
I20260620 18:43:06.912348 10296 tablet_replica.cc:330] T 8e116c8753d441948f655c57bfdf44f7 P a6126f70151449e798ad1e66f88c6efd: stopping tablet replica
I20260620 18:43:06.912904 10296 raft_consensus.cc:2229] T 8e116c8753d441948f655c57bfdf44f7 P a6126f70151449e798ad1e66f88c6efd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:43:06.913533 10296 raft_consensus.cc:2258] T 8e116c8753d441948f655c57bfdf44f7 P a6126f70151449e798ad1e66f88c6efd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:43:06.914000 9782 catalog_manager.cc:4841] TS 04390245bb3c49e8952a231e960f7ee5 (127.23.115.2:37151): tablet 8e116c8753d441948f655c57bfdf44f7 (table test-table [id=49f46a4671674230918117429e1746b0]) successfully deleted
I20260620 18:43:06.915741 10296 ts_tablet_manager.cc:1905] T 8e116c8753d441948f655c57bfdf44f7 P a6126f70151449e798ad1e66f88c6efd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:43:06.917375 10267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:06.892688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47622 (local address 127.23.115.1:36321)
0620 18:43:06.892874 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:06.892880 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:06.896304 (+ 3424us) server_negotiation.cc:407] Connection header received
0620 18:43:06.896596 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:06.896603 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:06.896696 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:06.896771 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:06.899617 (+ 2846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:06.900808 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:06.902896 (+ 2088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:06.903576 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:06.903677 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:06.909265 (+ 5588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:06.909286 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:06.909306 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:06.909354 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:06.913398 (+ 4044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:06.914344 (+ 946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:06.914354 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:06.914867 (+ 513us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:06.915075 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:06.915568 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:06.915574 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:06.915577 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:06.916048 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:06.916403 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:06.916729 (+ 326us) server_negotiation.cc:299] Negotiation successful
0620 18:43:06.916984 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 18:43:06.918079 9980 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:43:06.921577 9782 catalog_manager.cc:4841] TS 293e2d88c09146839377760014804070 (127.23.115.3:45923): tablet 8e116c8753d441948f655c57bfdf44f7 (table test-table [id=49f46a4671674230918117429e1746b0]) successfully deleted
I20260620 18:43:07.008285 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 9857
I20260620 18:43:07.029966 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 9991
W20260620 18:43:07.030033 9780 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.115.1:36321 (error 108)
I20260620 18:43:07.033032 10268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:07.032612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.1:36321 (local address 127.0.0.1:47638)
0620 18:43:07.032774 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:07.032865 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.1:36321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 18:43:07.033383 9780 catalog_manager.cc:4584] TS a6126f70151449e798ad1e66f88c6efd (127.23.115.1:36321): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8e116c8753d441948f655c57bfdf44f7: Network error: Client connection negotiation failed: client connection to 127.23.115.1:36321: connect: Connection refused (error 111)
Jun 20 18:43:07 dist-test-slave-d645 krb5kdc[9751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980984, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.115.2@KRBTEST.COM
I20260620 18:43:07.049077 10269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:07.030791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36698 (local address 127.23.115.2:37151)
0620 18:43:07.031030 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:07.031036 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:07.031120 (+ 84us) server_negotiation.cc:407] Connection header received
0620 18:43:07.031324 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:07.031330 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:07.031415 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:07.031523 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:07.033261 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:07.034240 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:07.036341 (+ 2101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:07.036739 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:07.036850 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:07.041949 (+ 5099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:07.041970 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:07.041990 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:07.042045 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:07.045777 (+ 3732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:07.046560 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:07.046571 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:07.046575 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:07.046742 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:07.047397 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:07.047402 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:07.047406 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:07.047904 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:07.048193 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:07.048471 (+ 278us) server_negotiation.cc:299] Negotiation successful
0620 18:43:07.048709 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 18:43:07.050063 10114 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:43:07.087142 10268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:07.086525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.1:36321 (local address 127.0.0.1:47652)
0620 18:43:07.086802 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:07.086920 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.1:36321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
I20260620 18:43:07.088290 10268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:07.087868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.1:36321 (local address 127.0.0.1:47664)
0620 18:43:07.088057 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:07.088131 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.1:36321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260620 18:43:07.143324 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 9991
I20260620 18:43:07.148931 10268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:07.148436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.1:36321 (local address 127.0.0.1:47666)
0620 18:43:07.148641 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:07.148743 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.1:36321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260620 18:43:07.150152 10268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:07.149658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.1:36321 (local address 127.0.0.1:47680)
0620 18:43:07.149835 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:07.149997 (+ 162us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.1:36321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260620 18:43:07.158550 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 10125
Jun 20 18:43:07 dist-test-slave-d645 krb5kdc[9751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980984, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.115.3@KRBTEST.COM
I20260620 18:43:07.176015 10266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:07.159364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53834 (local address 127.23.115.3:45923)
0620 18:43:07.159562 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:07.159568 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:07.159594 (+ 26us) server_negotiation.cc:407] Connection header received
0620 18:43:07.159659 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:07.159664 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:07.159744 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:07.159881 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:07.161148 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:07.162302 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:07.164074 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:07.164743 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:07.164902 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:07.169564 (+ 4662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:07.169576 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:07.169589 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:07.169624 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:07.172574 (+ 2950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:07.173453 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:07.173461 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:07.173463 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:07.173658 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:07.174130 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:07.174134 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:07.174137 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:07.174513 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:07.174765 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:07.175402 (+ 637us) server_negotiation.cc:299] Negotiation successful
0620 18:43:07.175664 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 18:43:07.176756 10248 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:43:07.231124 10268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:07.230540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.1:36321 (local address 127.0.0.1:47690)
0620 18:43:07.230793 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:07.230894 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.1:36321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260620 18:43:07.232266 10268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:07.231824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.1:36321 (local address 127.0.0.1:47696)
0620 18:43:07.232063 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:07.232134 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.1:36321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
I20260620 18:43:07.298681 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 10125
I20260620 18:43:07.315155 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 9764
I20260620 18:43:07.316308 9826 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:43:07.425560 10268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:07.425049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.1:36321 (local address 127.0.0.1:47708)
0620 18:43:07.425231 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:07.425346 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.1:36321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 18:43:07.425858 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 9764
2026-06-20T18:43:07Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4167 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-20T18:43:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:43:07Z Disabled control of system clock
I20260620 18:43:07.496229 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980676253068-24012-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980676253068-24012-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980676253068-24012-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.115.62:37791
--webserver_interface=127.23.115.62
--webserver_port=0
--builtin_ntp_servers=127.23.115.20:39931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.115.62:37791
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 18:43:07.819955 10306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:43:07.820230 10306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:43:07.820310 10306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:43:07.827546 10306 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:43:07.827669 10306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:43:07.827716 10306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:43:07.827764 10306 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:43:07.827808 10306 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:43:07.837129 10306 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:39931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980676253068-24012-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980676253068-24012-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.115.62:37791
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.115.62:37791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.115.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980676253068-24012-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:43:07.838340 10306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:43:07.840018 10306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:43:07.851719 10314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:07.851797 10312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:07.851951 10311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:07.852741 10306 server_base.cc:1034] running on GCE node
I20260620 18:43:07.853505 10306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:43:07.854924 10306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:43:07.856190 10306 hybrid_clock.cc:648] HybridClock initialized: now 1781980987856137 us; error 44 us; skew 500 ppm
I20260620 18:43:07.856642 10306 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:43:07.859575 10306 webserver.cc:460] Webserver started at http://127.23.115.62:45143/ using document root <none> and password file <none>
I20260620 18:43:07.860266 10306 fs_manager.cc:359] Metadata directory not provided
I20260620 18:43:07.860369 10306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:43:07.860682 10306 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:43:07.863972 10306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980676253068-24012-0/minicluster-data/master-0/data/instance:
uuid: "ed966b542890442288240d4737192b93"
format_stamp: "Formatted at 2026-06-20 18:43:07 on dist-test-slave-d645"
server_key: "2b411ea9cffc83d1f473f4fbf008c719"
server_key_iv: "cb1b6acd5291a3d696620bda35bf0ae9"
server_key_version: "encryptionkey@0"
I20260620 18:43:07.864849 10306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980676253068-24012-0/minicluster-data/master-0/wal/instance:
uuid: "ed966b542890442288240d4737192b93"
format_stamp: "Formatted at 2026-06-20 18:43:07 on dist-test-slave-d645"
server_key: "2b411ea9cffc83d1f473f4fbf008c719"
server_key_iv: "cb1b6acd5291a3d696620bda35bf0ae9"
server_key_version: "encryptionkey@0"
I20260620 18:43:07.870257 10306 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260620 18:43:07.874029 10320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:07.876012 10306 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 18:43:07.876168 10306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980676253068-24012-0/minicluster-data/master-0/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980676253068-24012-0/minicluster-data/master-0/wal
uuid: "ed966b542890442288240d4737192b93"
format_stamp: "Formatted at 2026-06-20 18:43:07 on dist-test-slave-d645"
server_key: "2b411ea9cffc83d1f473f4fbf008c719"
server_key_iv: "cb1b6acd5291a3d696620bda35bf0ae9"
server_key_version: "encryptionkey@0"
I20260620 18:43:07.876363 10306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980676253068-24012-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980676253068-24012-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980676253068-24012-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:43:07.889946 10306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:43:07.890730 10306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:43:07.891022 10306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:43:07.908447 10306 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.62:37791
I20260620 18:43:07.908480 10371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.62:37791 every 8 connection(s)
I20260620 18:43:07.909703 10306 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
I20260620 18:43:07.913244 10372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:43:07.918437 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 10306
I20260620 18:43:07.918603 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980676253068-24012-0/minicluster-data/master-0/wal/instance
I20260620 18:43:07.919210 24012 external_mini_cluster.cc:1428] Setting key 016b3483e5d6a9fbde59ded1da22ed33
I20260620 18:43:07.924510 10372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed966b542890442288240d4737192b93: Bootstrap starting.
I20260620 18:43:07.927928 10372 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed966b542890442288240d4737192b93: Neither blocks nor log segments found. Creating new log.
I20260620 18:43:07.929193 10372 log.cc:824] T 00000000000000000000000000000000 P ed966b542890442288240d4737192b93: Log is configured to *not* fsync() on all Append() calls
I20260620 18:43:07.932622 10372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed966b542890442288240d4737192b93: No bootstrap required, opened a new log
I20260620 18:43:07.934093 10375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:07.921132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54624 (local address 127.23.115.62:37791)
0620 18:43:07.922355 (+ 1223us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:07.922368 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:07.922415 (+ 47us) server_negotiation.cc:407] Connection header received
0620 18:43:07.924600 (+ 2185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:07.924626 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:07.925128 (+ 502us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:07.925464 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:07.926205 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:07.927568 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:07.929444 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:07.930439 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:07.930629 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:07.931878 (+ 1249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:07.931885 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:07.931899 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:43:07.932422 (+ 523us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:07.932501 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:43:07.932522 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:07.932619 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:07.932862 (+ 243us) server_negotiation.cc:299] Negotiation successful
0620 18:43:07.933164 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":406,"threads_started":1}
I20260620 18:43:07.940128 10372 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed966b542890442288240d4737192b93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed966b542890442288240d4737192b93" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 37791 } }
I20260620 18:43:07.940479 10372 raft_consensus.cc:374] T 00000000000000000000000000000000 P ed966b542890442288240d4737192b93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:43:07.940575 10372 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed966b542890442288240d4737192b93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed966b542890442288240d4737192b93, State: Initialized, Role: FOLLOWER
I20260620 18:43:07.941150 10372 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed966b542890442288240d4737192b93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed966b542890442288240d4737192b93" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 37791 } }
I20260620 18:43:07.941453 10372 raft_consensus.cc:388] T 00000000000000000000000000000000 P ed966b542890442288240d4737192b93 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:43:07.941591 10372 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed966b542890442288240d4737192b93 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:43:07.941748 10372 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed966b542890442288240d4737192b93 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:43:07.944013 10372 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed966b542890442288240d4737192b93 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed966b542890442288240d4737192b93" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 37791 } }
I20260620 18:43:07.944480 10372 leader_election.cc:302] T 00000000000000000000000000000000 P ed966b542890442288240d4737192b93 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed966b542890442288240d4737192b93; no voters:
I20260620 18:43:07.944994 10372 leader_election.cc:288] T 00000000000000000000000000000000 P ed966b542890442288240d4737192b93 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:43:07.945178 10377 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed966b542890442288240d4737192b93 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:43:07.945838 10377 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed966b542890442288240d4737192b93 [term 1 LEADER]: Becoming Leader. State: Replica: ed966b542890442288240d4737192b93, State: Running, Role: LEADER
I20260620 18:43:07.946314 10377 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed966b542890442288240d4737192b93 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed966b542890442288240d4737192b93" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 37791 } }
I20260620 18:43:07.947009 10372 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed966b542890442288240d4737192b93 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:43:07.948836 10379 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed966b542890442288240d4737192b93 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed966b542890442288240d4737192b93. Latest consensus state: current_term: 1 leader_uuid: "ed966b542890442288240d4737192b93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed966b542890442288240d4737192b93" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 37791 } } }
I20260620 18:43:07.948896 10378 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed966b542890442288240d4737192b93 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ed966b542890442288240d4737192b93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed966b542890442288240d4737192b93" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 37791 } } }
I20260620 18:43:07.949259 10379 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed966b542890442288240d4737192b93 [sys.catalog]: This master's current role is: LEADER
I20260620 18:43:07.949254 10378 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed966b542890442288240d4737192b93 [sys.catalog]: This master's current role is: LEADER
I20260620 18:43:07.952003 10384 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:43:07.956882 10384 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:43:07.966398 10384 catalog_manager.cc:1347] Generated new cluster ID: 709b86a004954900b189c3056c2b885b
I20260620 18:43:07.966516 10384 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:43:07.985152 10384 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:43:07.986464 10384 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:43:07.998843 10384 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ed966b542890442288240d4737192b93: Generated new TSK 0
I20260620 18:43:08.000003 10384 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:43:08.011576 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980676253068-24012-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980676253068-24012-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980676253068-24012-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980676253068-24012-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.115.1:0
--local_ip_for_outbound_sockets=127.23.115.1
--webserver_interface=127.23.115.1
--webserver_port=0
--tserver_master_addrs=127.23.115.62:37791
--builtin_ntp_servers=127.23.115.20:39931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 18:43:08.349951 10396 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:43:08.350258 10396 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:43:08.350358 10396 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:43:08.357841 10396 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:43:08.357964 10396 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:43:08.358086 10396 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.115.1
I20260620 18:43:08.367759 10396 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:39931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980676253068-24012-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980676253068-24012-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.115.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980676253068-24012-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.115.1
--webserver_port=0
--tserver_master_addrs=127.23.115.62:37791
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.115.1
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980676253068-24012-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:43:08.369000 10396 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:43:08.370702 10396 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:43:08.383183 10401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:08.383255 10402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:08.383284 10404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:08.384717 10396 server_base.cc:1034] running on GCE node
I20260620 18:43:08.385342 10396 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:43:08.386420 10396 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:43:08.387655 10396 hybrid_clock.cc:648] HybridClock initialized: now 1781980988387604 us; error 42 us; skew 500 ppm
I20260620 18:43:08.387989 10396 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:43:08.390647 10396 webserver.cc:460] Webserver started at http://127.23.115.1:35579/ using document root <none> and password file <none>
I20260620 18:43:08.391410 10396 fs_manager.cc:359] Metadata directory not provided
I20260620 18:43:08.391510 10396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:43:08.391813 10396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:43:08.394752 10396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980676253068-24012-0/minicluster-data/ts-0/data/instance:
uuid: "effdc8a6cb9a4b4daad6a2eef0ce7c42"
format_stamp: "Formatted at 2026-06-20 18:43:08 on dist-test-slave-d645"
server_key: "537cfcae57ac52cda515c04f64f24962"
server_key_iv: "c3f7e7529c4f4659253b93f5846622b0"
server_key_version: "encryptionkey@0"
I20260620 18:43:08.395567 10396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980676253068-24012-0/minicluster-data/ts-0/wal/instance:
uuid: "effdc8a6cb9a4b4daad6a2eef0ce7c42"
format_stamp: "Formatted at 2026-06-20 18:43:08 on dist-test-slave-d645"
server_key: "537cfcae57ac52cda515c04f64f24962"
server_key_iv: "c3f7e7529c4f4659253b93f5846622b0"
server_key_version: "encryptionkey@0"
I20260620 18:43:08.401068 10396 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:43:08.405139 10410 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:08.406605 10396 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:43:08.406765 10396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980676253068-24012-0/minicluster-data/ts-0/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980676253068-24012-0/minicluster-data/ts-0/wal
uuid: "effdc8a6cb9a4b4daad6a2eef0ce7c42"
format_stamp: "Formatted at 2026-06-20 18:43:08 on dist-test-slave-d645"
server_key: "537cfcae57ac52cda515c04f64f24962"
server_key_iv: "c3f7e7529c4f4659253b93f5846622b0"
server_key_version: "encryptionkey@0"
I20260620 18:43:08.406998 10396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980676253068-24012-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980676253068-24012-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980676253068-24012-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:43:08.427862 10396 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:43:08.428707 10396 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:43:08.429008 10396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:43:08.429991 10396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:43:08.431504 10396 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:43:08.431597 10396 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:08.431701 10396 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:43:08.431769 10396 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:08.463086 10396 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.1:39445
I20260620 18:43:08.463155 10522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.1:39445 every 8 connection(s)
I20260620 18:43:08.464365 10396 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980676253068-24012-0/minicluster-data/ts-0/data/info.pb
I20260620 18:43:08.467965 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 10396
I20260620 18:43:08.468147 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980676253068-24012-0/minicluster-data/ts-0/wal/instance
I20260620 18:43:08.468627 24012 external_mini_cluster.cc:1428] Setting key 7956d6847d8678e78f3fea654ed86348
I20260620 18:43:08.471267 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980676253068-24012-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980676253068-24012-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980676253068-24012-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980676253068-24012-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.115.2:0
--local_ip_for_outbound_sockets=127.23.115.2
--webserver_interface=127.23.115.2
--webserver_port=0
--tserver_master_addrs=127.23.115.62:37791
--builtin_ntp_servers=127.23.115.20:39931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 18:43:08.477167 10525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:08.466733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.1:39073 (local address 127.23.115.62:37791)
0620 18:43:08.467212 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:08.467218 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:08.468387 (+ 1169us) server_negotiation.cc:407] Connection header received
0620 18:43:08.470271 (+ 1884us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:08.470275 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:08.470335 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:08.470426 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:08.471515 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:08.472220 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:08.474280 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:08.474818 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:08.474978 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:08.476441 (+ 1463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:08.476446 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:08.476450 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:43:08.476468 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:08.476512 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:43:08.476524 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:08.476597 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:08.476804 (+ 207us) server_negotiation.cc:299] Negotiation successful
0620 18:43:08.476890 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":237,"threads_started":1}
I20260620 18:43:08.478366 10526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:08.467307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:37791 (local address 127.23.115.1:39073)
0620 18:43:08.468201 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:08.468278 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:08.470017 (+ 1739us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:08.470623 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:08.470631 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:08.470715 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:43:08.471322 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:08.471343 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:08.472382 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:08.472388 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:08.474107 (+ 1719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:08.474121 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:08.475059 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:08.475065 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:08.475337 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:08.476075 (+ 738us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:43:08.476079 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:43:08.476132 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:43:08.476238 (+ 106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:43:08.476611 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:43:08.476619 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:43:08.476627 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 18:43:08.476786 (+ 159us) client_negotiation.cc:240] Negotiation successful
0620 18:43:08.477161 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":263,"threads_started":1}
I20260620 18:43:08.480007 10523 heartbeater.cc:347] Connected to a master server at 127.23.115.62:37791
I20260620 18:43:08.480301 10523 heartbeater.cc:464] Registering TS with master...
I20260620 18:43:08.481017 10523 heartbeater.cc:511] Master 127.23.115.62:37791 requested a full tablet report, sending...
I20260620 18:43:08.483012 10337 ts_manager.cc:194] Registered new tserver with Master: effdc8a6cb9a4b4daad6a2eef0ce7c42 (127.23.115.1:39445)
I20260620 18:43:08.484514 10337 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.115.1:39073
W20260620 18:43:08.794860 10527 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:43:08.795305 10527 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:43:08.795449 10527 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:43:08.802778 10527 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:43:08.802891 10527 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:43:08.803071 10527 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.115.2
I20260620 18:43:08.812474 10527 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:39931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980676253068-24012-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980676253068-24012-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.115.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980676253068-24012-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.115.2
--webserver_port=0
--tserver_master_addrs=127.23.115.62:37791
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.115.2
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980676253068-24012-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:43:08.813740 10527 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:43:08.815493 10527 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 18:43:08.829591 10527 server_base.cc:1034] running on GCE node
W20260620 18:43:08.829551 10533 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:08.829676 10532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:08.829732 10535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:08.830637 10527 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:43:08.831641 10527 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:43:08.832871 10527 hybrid_clock.cc:648] HybridClock initialized: now 1781980988832768 us; error 94 us; skew 500 ppm
I20260620 18:43:08.833256 10527 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:43:08.835641 10527 webserver.cc:460] Webserver started at http://127.23.115.2:35541/ using document root <none> and password file <none>
I20260620 18:43:08.836313 10527 fs_manager.cc:359] Metadata directory not provided
I20260620 18:43:08.836413 10527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:43:08.836721 10527 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:43:08.839558 10527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980676253068-24012-0/minicluster-data/ts-1/data/instance:
uuid: "a3a2e72dd62d45e19ecba5c9b37fb8d8"
format_stamp: "Formatted at 2026-06-20 18:43:08 on dist-test-slave-d645"
server_key: "417055cb59df30af952b68cafb904eca"
server_key_iv: "6be1ea820d27a6c1fbc5e36ecf67eb74"
server_key_version: "encryptionkey@0"
I20260620 18:43:08.840301 10527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980676253068-24012-0/minicluster-data/ts-1/wal/instance:
uuid: "a3a2e72dd62d45e19ecba5c9b37fb8d8"
format_stamp: "Formatted at 2026-06-20 18:43:08 on dist-test-slave-d645"
server_key: "417055cb59df30af952b68cafb904eca"
server_key_iv: "6be1ea820d27a6c1fbc5e36ecf67eb74"
server_key_version: "encryptionkey@0"
I20260620 18:43:08.845274 10527 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:43:08.849431 10541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:08.850984 10527 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:43:08.851158 10527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980676253068-24012-0/minicluster-data/ts-1/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980676253068-24012-0/minicluster-data/ts-1/wal
uuid: "a3a2e72dd62d45e19ecba5c9b37fb8d8"
format_stamp: "Formatted at 2026-06-20 18:43:08 on dist-test-slave-d645"
server_key: "417055cb59df30af952b68cafb904eca"
server_key_iv: "6be1ea820d27a6c1fbc5e36ecf67eb74"
server_key_version: "encryptionkey@0"
I20260620 18:43:08.851418 10527 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980676253068-24012-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980676253068-24012-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980676253068-24012-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:43:08.872531 10527 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:43:08.873354 10527 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:43:08.873632 10527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:43:08.874611 10527 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:43:08.876055 10527 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:43:08.876159 10527 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:08.876268 10527 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:43:08.876335 10527 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:08.907119 10527 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.2:38733
I20260620 18:43:08.907182 10653 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.2:38733 every 8 connection(s)
I20260620 18:43:08.908603 10527 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980676253068-24012-0/minicluster-data/ts-1/data/info.pb
I20260620 18:43:08.918285 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 10527
I20260620 18:43:08.918470 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980676253068-24012-0/minicluster-data/ts-1/wal/instance
I20260620 18:43:08.919005 24012 external_mini_cluster.cc:1428] Setting key 6b5a7fe173f51a85bf0142e0d1ba64e0
I20260620 18:43:08.921891 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980676253068-24012-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980676253068-24012-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980676253068-24012-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980676253068-24012-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.115.3:0
--local_ip_for_outbound_sockets=127.23.115.3
--webserver_interface=127.23.115.3
--webserver_port=0
--tserver_master_addrs=127.23.115.62:37791
--builtin_ntp_servers=127.23.115.20:39931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 18:43:08.922557 10525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:08.911349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.2:40625 (local address 127.23.115.62:37791)
0620 18:43:08.911523 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:08.911527 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:08.913585 (+ 2058us) server_negotiation.cc:407] Connection header received
0620 18:43:08.915519 (+ 1934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:08.915525 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:08.915611 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:08.915697 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:08.916714 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:08.917455 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:08.919470 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:08.920028 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:08.920162 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:08.921684 (+ 1522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:08.921691 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:08.921696 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:43:08.921718 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:08.921758 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:43:08.921775 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:08.921874 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:08.922130 (+ 256us) server_negotiation.cc:299] Negotiation successful
0620 18:43:08.922239 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 18:43:08.924700 10656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:08.911928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:37791 (local address 127.23.115.2:40625)
0620 18:43:08.913269 (+ 1341us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:08.913344 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:08.915245 (+ 1901us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:08.915928 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:08.915936 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:08.916023 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:43:08.916560 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:08.916578 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:08.917606 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:08.917610 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:08.919279 (+ 1669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:08.919297 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:08.920257 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:08.920264 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:08.920555 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:08.921319 (+ 764us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:43:08.921324 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:43:08.921382 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:43:08.921484 (+ 102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:43:08.921878 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:43:08.921888 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:43:08.921896 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 18:43:08.922814 (+ 918us) client_negotiation.cc:240] Negotiation successful
0620 18:43:08.923301 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":333,"threads_started":1}
I20260620 18:43:08.926190 10654 heartbeater.cc:347] Connected to a master server at 127.23.115.62:37791
I20260620 18:43:08.926528 10654 heartbeater.cc:464] Registering TS with master...
I20260620 18:43:08.927451 10654 heartbeater.cc:511] Master 127.23.115.62:37791 requested a full tablet report, sending...
I20260620 18:43:08.929021 10337 ts_manager.cc:194] Registered new tserver with Master: a3a2e72dd62d45e19ecba5c9b37fb8d8 (127.23.115.2:38733)
I20260620 18:43:08.930058 10337 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.115.2:40625
W20260620 18:43:09.243093 10657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:43:09.243373 10657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:43:09.243458 10657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:43:09.251001 10657 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:43:09.251103 10657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:43:09.251238 10657 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.115.3
I20260620 18:43:09.261258 10657 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:39931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980676253068-24012-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980676253068-24012-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.115.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980676253068-24012-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.115.3
--webserver_port=0
--tserver_master_addrs=127.23.115.62:37791
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.115.3
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980676253068-24012-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:43:09.262503 10657 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:43:09.264261 10657 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:43:09.277881 10662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:09.277982 10663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:09.277956 10665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:09.278529 10657 server_base.cc:1034] running on GCE node
I20260620 18:43:09.279201 10657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:43:09.280192 10657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:43:09.281414 10657 hybrid_clock.cc:648] HybridClock initialized: now 1781980989281358 us; error 47 us; skew 500 ppm
I20260620 18:43:09.281754 10657 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:43:09.284072 10657 webserver.cc:460] Webserver started at http://127.23.115.3:40419/ using document root <none> and password file <none>
I20260620 18:43:09.284799 10657 fs_manager.cc:359] Metadata directory not provided
I20260620 18:43:09.284906 10657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:43:09.285198 10657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:43:09.288179 10657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980676253068-24012-0/minicluster-data/ts-2/data/instance:
uuid: "11912959fa6a4676a568298b7da1b71d"
format_stamp: "Formatted at 2026-06-20 18:43:09 on dist-test-slave-d645"
server_key: "335f5129b3ad48ac98aa08301a5bd3e4"
server_key_iv: "c0583ab7a7806ed317bf62293224893e"
server_key_version: "encryptionkey@0"
I20260620 18:43:09.288957 10657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980676253068-24012-0/minicluster-data/ts-2/wal/instance:
uuid: "11912959fa6a4676a568298b7da1b71d"
format_stamp: "Formatted at 2026-06-20 18:43:09 on dist-test-slave-d645"
server_key: "335f5129b3ad48ac98aa08301a5bd3e4"
server_key_iv: "c0583ab7a7806ed317bf62293224893e"
server_key_version: "encryptionkey@0"
I20260620 18:43:09.293953 10657 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 18:43:09.297658 10671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:09.298924 10657 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:43:09.299129 10657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980676253068-24012-0/minicluster-data/ts-2/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980676253068-24012-0/minicluster-data/ts-2/wal
uuid: "11912959fa6a4676a568298b7da1b71d"
format_stamp: "Formatted at 2026-06-20 18:43:09 on dist-test-slave-d645"
server_key: "335f5129b3ad48ac98aa08301a5bd3e4"
server_key_iv: "c0583ab7a7806ed317bf62293224893e"
server_key_version: "encryptionkey@0"
I20260620 18:43:09.299329 10657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980676253068-24012-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980676253068-24012-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980676253068-24012-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:43:09.323844 10657 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:43:09.324748 10657 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:43:09.325034 10657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:43:09.326637 10657 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:43:09.328702 10657 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:43:09.328819 10657 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:09.328930 10657 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:43:09.329000 10657 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:09.363615 10657 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.3:37319
I20260620 18:43:09.363700 10783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.3:37319 every 8 connection(s)
I20260620 18:43:09.364857 10657 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980676253068-24012-0/minicluster-data/ts-2/data/info.pb
I20260620 18:43:09.368506 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 10657
I20260620 18:43:09.368700 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980676253068-24012-0/minicluster-data/ts-2/wal/instance
I20260620 18:43:09.369172 24012 external_mini_cluster.cc:1428] Setting key 19757b0399876286b280221a3071f9ce
I20260620 18:43:09.377280 10525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:09.367365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.3:44151 (local address 127.23.115.62:37791)
0620 18:43:09.367530 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:09.367534 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:09.368889 (+ 1355us) server_negotiation.cc:407] Connection header received
0620 18:43:09.370826 (+ 1937us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:09.370832 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:09.370924 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:09.371042 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:09.372149 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:09.372881 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:09.374397 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:09.374808 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:09.374913 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:09.376475 (+ 1562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:09.376481 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:09.376484 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:43:09.376503 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:09.376534 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:43:09.376547 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:09.376620 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:09.376857 (+ 237us) server_negotiation.cc:299] Negotiation successful
0620 18:43:09.376955 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 18:43:09.378298 10786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:09.367752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:37791 (local address 127.23.115.3:44151)
0620 18:43:09.368708 (+ 956us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:09.368758 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:09.370544 (+ 1786us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:09.371279 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:09.371287 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:09.371377 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:43:09.371996 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:09.372016 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:09.373030 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:09.373034 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:09.374262 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:09.374274 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:09.375040 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:09.375048 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:09.375274 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:09.376115 (+ 841us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:43:09.376118 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:43:09.376158 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:43:09.376274 (+ 116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:43:09.376638 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:43:09.376648 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:43:09.376656 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 18:43:09.376832 (+ 176us) client_negotiation.cc:240] Negotiation successful
0620 18:43:09.377244 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":295,"threads_started":1}
I20260620 18:43:09.379756 10784 heartbeater.cc:347] Connected to a master server at 127.23.115.62:37791
I20260620 18:43:09.380056 10784 heartbeater.cc:464] Registering TS with master...
I20260620 18:43:09.380841 10784 heartbeater.cc:511] Master 127.23.115.62:37791 requested a full tablet report, sending...
I20260620 18:43:09.382424 10337 ts_manager.cc:194] Registered new tserver with Master: 11912959fa6a4676a568298b7da1b71d (127.23.115.3:37319)
I20260620 18:43:09.383420 10337 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.115.3:44151
I20260620 18:43:09.384276 24012 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:43:09.394482 10525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:09.388271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54640 (local address 127.23.115.62:37791)
0620 18:43:09.388421 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:09.388428 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:09.388814 (+ 386us) server_negotiation.cc:407] Connection header received
0620 18:43:09.388997 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:09.389003 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:09.389080 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:09.389161 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:09.389616 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:09.390314 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:09.391632 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:09.392039 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:09.392145 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:09.393761 (+ 1616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:09.393767 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:09.393771 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:43:09.393790 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:09.393824 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:43:09.393837 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:09.393956 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:09.394152 (+ 196us) server_negotiation.cc:299] Negotiation successful
0620 18:43:09.394208 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 18:43:09.399849 10336 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54640:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:43:09.407588 10336 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:43:09.433422 10797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:09.421509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.2:38733 (local address 127.0.0.1:34504)
0620 18:43:09.424306 (+ 2797us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:09.424328 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:09.424453 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:09.425315 (+ 862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:09.425323 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:09.425343 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:43:09.426057 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:09.426069 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:09.428072 (+ 2003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:09.428078 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:09.430733 (+ 2655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:09.430748 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:09.432442 (+ 1694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:09.432449 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:09.432647 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:09.432704 (+ 57us) client_negotiation.cc:769] Sending connection context
0620 18:43:09.432819 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 18:43:09.433028 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2680,"mutex_wait_us":75,"thread_start_us":2847,"threads_started":1}
I20260620 18:43:09.433409 10799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:09.422243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34504 (local address 127.23.115.2:38733)
0620 18:43:09.424016 (+ 1773us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:09.424025 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:09.424417 (+ 392us) server_negotiation.cc:407] Connection header received
0620 18:43:09.424599 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:09.424606 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:09.424876 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:09.425046 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:43:09.426255 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:09.427880 (+ 1625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:09.430923 (+ 3043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:09.432262 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:09.432378 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:09.432641 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:09.432876 (+ 235us) server_negotiation.cc:299] Negotiation successful
0620 18:43:09.433047 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1535,"spinlock_wait_cycles":23424,"thread_start_us":1491,"threads_started":1}
I20260620 18:43:09.433794 10796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:09.421138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.3:37319 (local address 127.0.0.1:41500)
0620 18:43:09.422368 (+ 1230us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:09.422415 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:09.422619 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:09.425094 (+ 2475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:09.425103 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:09.425132 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:43:09.425614 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:09.425628 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:09.428093 (+ 2465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:09.428097 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:09.431059 (+ 2962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:09.431073 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:09.433017 (+ 1944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:09.433023 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:09.433198 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:09.433232 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 18:43:09.433320 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 18:43:09.433458 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":916,"mutex_wait_us":339,"thread_start_us":1297,"threads_started":1}
I20260620 18:43:09.434777 10798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:09.421418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41500 (local address 127.23.115.3:37319)
0620 18:43:09.424479 (+ 3061us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:09.424484 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:09.424506 (+ 22us) server_negotiation.cc:407] Connection header received
0620 18:43:09.424599 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:09.424606 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:09.424857 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:09.425857 (+ 1000us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:43:09.426133 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:09.427959 (+ 1826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:09.431259 (+ 3300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:09.432519 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:09.432641 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:09.432890 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:09.434254 (+ 1364us) server_negotiation.cc:299] Negotiation successful
0620 18:43:09.434426 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2925,"thread_start_us":2775,"threads_started":1}
I20260620 18:43:09.437562 10589 tablet_service.cc:1467] Processing CreateTablet for tablet a1b9c1c4fbab46778c7cc36f8112f6fb (DEFAULT_TABLE table=test-table [id=369c6c2a27514010aa4a8b74dc55f543]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:43:09.437785 10719 tablet_service.cc:1467] Processing CreateTablet for tablet a1b9c1c4fbab46778c7cc36f8112f6fb (DEFAULT_TABLE table=test-table [id=369c6c2a27514010aa4a8b74dc55f543]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:43:09.439108 10719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1b9c1c4fbab46778c7cc36f8112f6fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:43:09.439110 10589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1b9c1c4fbab46778c7cc36f8112f6fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:43:09.439647 10801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:09.425000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42122 (local address 127.23.115.1:39445)
0620 18:43:09.431260 (+ 6260us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:09.431268 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:09.431299 (+ 31us) server_negotiation.cc:407] Connection header received
0620 18:43:09.431397 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:09.431405 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:09.431672 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:09.431870 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:43:09.432475 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:09.433893 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:09.436822 (+ 2929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:09.438202 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:09.438333 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:09.438563 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:09.439165 (+ 602us) server_negotiation.cc:299] Negotiation successful
0620 18:43:09.439329 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6088,"thread_start_us":5968,"threads_started":1}
I20260620 18:43:09.439983 10800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:09.424694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.1:39445 (local address 127.0.0.1:42122)
0620 18:43:09.429805 (+ 5111us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:09.429831 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:09.429946 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:09.431883 (+ 1937us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:09.431888 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:09.431915 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:43:09.432268 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:09.432280 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:09.434054 (+ 1774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:09.434059 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:09.436597 (+ 2538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:09.436613 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:09.438507 (+ 1894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:09.438515 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:09.438675 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:09.438875 (+ 200us) client_negotiation.cc:769] Sending connection context
0620 18:43:09.439108 (+ 233us) client_negotiation.cc:240] Negotiation successful
0620 18:43:09.439532 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4981,"thread_start_us":4834,"threads_started":1}
I20260620 18:43:09.443961 10458 tablet_service.cc:1467] Processing CreateTablet for tablet a1b9c1c4fbab46778c7cc36f8112f6fb (DEFAULT_TABLE table=test-table [id=369c6c2a27514010aa4a8b74dc55f543]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:43:09.445425 10458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1b9c1c4fbab46778c7cc36f8112f6fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:43:09.450677 10802 tablet_bootstrap.cc:492] T a1b9c1c4fbab46778c7cc36f8112f6fb P a3a2e72dd62d45e19ecba5c9b37fb8d8: Bootstrap starting.
I20260620 18:43:09.454433 10802 tablet_bootstrap.cc:654] T a1b9c1c4fbab46778c7cc36f8112f6fb P a3a2e72dd62d45e19ecba5c9b37fb8d8: Neither blocks nor log segments found. Creating new log.
I20260620 18:43:09.455999 10802 log.cc:824] T a1b9c1c4fbab46778c7cc36f8112f6fb P a3a2e72dd62d45e19ecba5c9b37fb8d8: Log is configured to *not* fsync() on all Append() calls
I20260620 18:43:09.457818 10803 tablet_bootstrap.cc:492] T a1b9c1c4fbab46778c7cc36f8112f6fb P 11912959fa6a4676a568298b7da1b71d: Bootstrap starting.
I20260620 18:43:09.460079 10805 tablet_bootstrap.cc:492] T a1b9c1c4fbab46778c7cc36f8112f6fb P effdc8a6cb9a4b4daad6a2eef0ce7c42: Bootstrap starting.
I20260620 18:43:09.460397 10802 tablet_bootstrap.cc:492] T a1b9c1c4fbab46778c7cc36f8112f6fb P a3a2e72dd62d45e19ecba5c9b37fb8d8: No bootstrap required, opened a new log
I20260620 18:43:09.460772 10802 ts_tablet_manager.cc:1397] T a1b9c1c4fbab46778c7cc36f8112f6fb P a3a2e72dd62d45e19ecba5c9b37fb8d8: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260620 18:43:09.463469 10803 tablet_bootstrap.cc:654] T a1b9c1c4fbab46778c7cc36f8112f6fb P 11912959fa6a4676a568298b7da1b71d: Neither blocks nor log segments found. Creating new log.
I20260620 18:43:09.464112 10805 tablet_bootstrap.cc:654] T a1b9c1c4fbab46778c7cc36f8112f6fb P effdc8a6cb9a4b4daad6a2eef0ce7c42: Neither blocks nor log segments found. Creating new log.
I20260620 18:43:09.465131 10803 log.cc:824] T a1b9c1c4fbab46778c7cc36f8112f6fb P 11912959fa6a4676a568298b7da1b71d: Log is configured to *not* fsync() on all Append() calls
I20260620 18:43:09.466068 10805 log.cc:824] T a1b9c1c4fbab46778c7cc36f8112f6fb P effdc8a6cb9a4b4daad6a2eef0ce7c42: Log is configured to *not* fsync() on all Append() calls
I20260620 18:43:09.468874 10803 tablet_bootstrap.cc:492] T a1b9c1c4fbab46778c7cc36f8112f6fb P 11912959fa6a4676a568298b7da1b71d: No bootstrap required, opened a new log
I20260620 18:43:09.469244 10803 ts_tablet_manager.cc:1397] T a1b9c1c4fbab46778c7cc36f8112f6fb P 11912959fa6a4676a568298b7da1b71d: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.005s
I20260620 18:43:09.475507 10802 raft_consensus.cc:348] T a1b9c1c4fbab46778c7cc36f8112f6fb P a3a2e72dd62d45e19ecba5c9b37fb8d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11912959fa6a4676a568298b7da1b71d" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 37319 } } peers { permanent_uuid: "a3a2e72dd62d45e19ecba5c9b37fb8d8" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 38733 } } peers { permanent_uuid: "effdc8a6cb9a4b4daad6a2eef0ce7c42" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 39445 } }
I20260620 18:43:09.476233 10802 raft_consensus.cc:374] T a1b9c1c4fbab46778c7cc36f8112f6fb P a3a2e72dd62d45e19ecba5c9b37fb8d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:43:09.476342 10802 raft_consensus.cc:729] T a1b9c1c4fbab46778c7cc36f8112f6fb P a3a2e72dd62d45e19ecba5c9b37fb8d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3a2e72dd62d45e19ecba5c9b37fb8d8, State: Initialized, Role: FOLLOWER
I20260620 18:43:09.476773 10805 tablet_bootstrap.cc:492] T a1b9c1c4fbab46778c7cc36f8112f6fb P effdc8a6cb9a4b4daad6a2eef0ce7c42: No bootstrap required, opened a new log
I20260620 18:43:09.477068 10805 ts_tablet_manager.cc:1397] T a1b9c1c4fbab46778c7cc36f8112f6fb P effdc8a6cb9a4b4daad6a2eef0ce7c42: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.010s
I20260620 18:43:09.476998 10802 consensus_queue.cc:260] T a1b9c1c4fbab46778c7cc36f8112f6fb P a3a2e72dd62d45e19ecba5c9b37fb8d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11912959fa6a4676a568298b7da1b71d" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 37319 } } peers { permanent_uuid: "a3a2e72dd62d45e19ecba5c9b37fb8d8" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 38733 } } peers { permanent_uuid: "effdc8a6cb9a4b4daad6a2eef0ce7c42" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 39445 } }
I20260620 18:43:09.478111 10654 heartbeater.cc:503] Master 127.23.115.62:37791 was elected leader, sending a full tablet report...
I20260620 18:43:09.480154 10802 ts_tablet_manager.cc:1428] T a1b9c1c4fbab46778c7cc36f8112f6fb P a3a2e72dd62d45e19ecba5c9b37fb8d8: Time spent starting tablet: real 0.019s user 0.009s sys 0.005s
I20260620 18:43:09.480466 10803 raft_consensus.cc:348] T a1b9c1c4fbab46778c7cc36f8112f6fb P 11912959fa6a4676a568298b7da1b71d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11912959fa6a4676a568298b7da1b71d" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 37319 } } peers { permanent_uuid: "a3a2e72dd62d45e19ecba5c9b37fb8d8" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 38733 } } peers { permanent_uuid: "effdc8a6cb9a4b4daad6a2eef0ce7c42" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 39445 } }
I20260620 18:43:09.480902 10803 raft_consensus.cc:374] T a1b9c1c4fbab46778c7cc36f8112f6fb P 11912959fa6a4676a568298b7da1b71d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:43:09.480996 10803 raft_consensus.cc:729] T a1b9c1c4fbab46778c7cc36f8112f6fb P 11912959fa6a4676a568298b7da1b71d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11912959fa6a4676a568298b7da1b71d, State: Initialized, Role: FOLLOWER
I20260620 18:43:09.481580 10803 consensus_queue.cc:260] T a1b9c1c4fbab46778c7cc36f8112f6fb P 11912959fa6a4676a568298b7da1b71d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11912959fa6a4676a568298b7da1b71d" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 37319 } } peers { permanent_uuid: "a3a2e72dd62d45e19ecba5c9b37fb8d8" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 38733 } } peers { permanent_uuid: "effdc8a6cb9a4b4daad6a2eef0ce7c42" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 39445 } }
I20260620 18:43:09.482780 10784 heartbeater.cc:503] Master 127.23.115.62:37791 was elected leader, sending a full tablet report...
I20260620 18:43:09.483485 10803 ts_tablet_manager.cc:1428] T a1b9c1c4fbab46778c7cc36f8112f6fb P 11912959fa6a4676a568298b7da1b71d: Time spent starting tablet: real 0.014s user 0.011s sys 0.000s
I20260620 18:43:09.486672 10805 raft_consensus.cc:348] T a1b9c1c4fbab46778c7cc36f8112f6fb P effdc8a6cb9a4b4daad6a2eef0ce7c42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11912959fa6a4676a568298b7da1b71d" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 37319 } } peers { permanent_uuid: "a3a2e72dd62d45e19ecba5c9b37fb8d8" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 38733 } } peers { permanent_uuid: "effdc8a6cb9a4b4daad6a2eef0ce7c42" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 39445 } }
I20260620 18:43:09.487154 10805 raft_consensus.cc:374] T a1b9c1c4fbab46778c7cc36f8112f6fb P effdc8a6cb9a4b4daad6a2eef0ce7c42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:43:09.487319 10805 raft_consensus.cc:729] T a1b9c1c4fbab46778c7cc36f8112f6fb P effdc8a6cb9a4b4daad6a2eef0ce7c42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: effdc8a6cb9a4b4daad6a2eef0ce7c42, State: Initialized, Role: FOLLOWER
I20260620 18:43:09.487432 10523 heartbeater.cc:503] Master 127.23.115.62:37791 was elected leader, sending a full tablet report...
I20260620 18:43:09.487996 10805 consensus_queue.cc:260] T a1b9c1c4fbab46778c7cc36f8112f6fb P effdc8a6cb9a4b4daad6a2eef0ce7c42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11912959fa6a4676a568298b7da1b71d" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 37319 } } peers { permanent_uuid: "a3a2e72dd62d45e19ecba5c9b37fb8d8" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 38733 } } peers { permanent_uuid: "effdc8a6cb9a4b4daad6a2eef0ce7c42" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 39445 } }
I20260620 18:43:09.489794 10805 ts_tablet_manager.cc:1428] T a1b9c1c4fbab46778c7cc36f8112f6fb P effdc8a6cb9a4b4daad6a2eef0ce7c42: Time spent starting tablet: real 0.013s user 0.010s sys 0.003s
I20260620 18:43:09.570567 10809 raft_consensus.cc:482] T a1b9c1c4fbab46778c7cc36f8112f6fb P 11912959fa6a4676a568298b7da1b71d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:43:09.570852 10809 raft_consensus.cc:504] T a1b9c1c4fbab46778c7cc36f8112f6fb P 11912959fa6a4676a568298b7da1b71d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11912959fa6a4676a568298b7da1b71d" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 37319 } } peers { permanent_uuid: "a3a2e72dd62d45e19ecba5c9b37fb8d8" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 38733 } } peers { permanent_uuid: "effdc8a6cb9a4b4daad6a2eef0ce7c42" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 39445 } }
I20260620 18:43:09.572499 10809 leader_election.cc:288] T a1b9c1c4fbab46778c7cc36f8112f6fb P 11912959fa6a4676a568298b7da1b71d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a3a2e72dd62d45e19ecba5c9b37fb8d8 (127.23.115.2:38733), effdc8a6cb9a4b4daad6a2eef0ce7c42 (127.23.115.1:39445)
I20260620 18:43:09.577800 10786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:09.572761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.2:38733 (local address 127.23.115.3:36613)
0620 18:43:09.572925 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:09.572944 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:09.573063 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:09.573590 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:09.573595 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:09.573622 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:43:09.573801 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:09.573811 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:09.574755 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:09.574759 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:09.576090 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:09.576102 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:09.577101 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:09.577106 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:09.577273 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:09.577296 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 18:43:09.577374 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 18:43:09.577508 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 18:43:09.577787 10799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:09.572983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.3:36613 (local address 127.23.115.2:38733)
0620 18:43:09.573231 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:09.573237 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:09.573257 (+ 20us) server_negotiation.cc:407] Connection header received
0620 18:43:09.573328 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:09.573333 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:09.573415 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:09.573516 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:43:09.573950 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:09.574629 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:09.576242 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:09.576964 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:09.577096 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:09.577178 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:09.577414 (+ 236us) server_negotiation.cc:299] Negotiation successful
0620 18:43:09.577532 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260620 18:43:09.578440 10801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:09.573292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.3:56749 (local address 127.23.115.1:39445)
0620 18:43:09.573465 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:09.573470 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:09.573796 (+ 326us) server_negotiation.cc:407] Connection header received
0620 18:43:09.573964 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:09.573967 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:09.574025 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:09.574095 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:43:09.574603 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:09.575300 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:09.576890 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:09.577624 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:09.577751 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:09.577813 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:09.578085 (+ 272us) server_negotiation.cc:299] Negotiation successful
0620 18:43:09.578207 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 18:43:09.578465 10811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:09.573104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.1:39445 (local address 127.23.115.3:56749)
0620 18:43:09.573696 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:09.573709 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:09.573819 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:09.574138 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:09.574142 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:09.574164 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:43:09.574461 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:09.574469 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:09.575432 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:09.575436 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:09.576718 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:09.576729 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:09.577772 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:09.577781 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:09.577949 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:09.577978 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 18:43:09.578055 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 18:43:09.578183 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":293,"threads_started":1}
I20260620 18:43:09.578744 10609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1b9c1c4fbab46778c7cc36f8112f6fb" candidate_uuid: "11912959fa6a4676a568298b7da1b71d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3a2e72dd62d45e19ecba5c9b37fb8d8" is_pre_election: true
I20260620 18:43:09.579336 10609 raft_consensus.cc:2454] T a1b9c1c4fbab46778c7cc36f8112f6fb P a3a2e72dd62d45e19ecba5c9b37fb8d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11912959fa6a4676a568298b7da1b71d in term 0.
I20260620 18:43:09.579382 10478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1b9c1c4fbab46778c7cc36f8112f6fb" candidate_uuid: "11912959fa6a4676a568298b7da1b71d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "effdc8a6cb9a4b4daad6a2eef0ce7c42" is_pre_election: true
I20260620 18:43:09.579891 10478 raft_consensus.cc:2454] T a1b9c1c4fbab46778c7cc36f8112f6fb P effdc8a6cb9a4b4daad6a2eef0ce7c42 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11912959fa6a4676a568298b7da1b71d in term 0.
I20260620 18:43:09.580060 10675 leader_election.cc:302] T a1b9c1c4fbab46778c7cc36f8112f6fb P 11912959fa6a4676a568298b7da1b71d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11912959fa6a4676a568298b7da1b71d, a3a2e72dd62d45e19ecba5c9b37fb8d8; no voters:
I20260620 18:43:09.580427 10809 raft_consensus.cc:2781] T a1b9c1c4fbab46778c7cc36f8112f6fb P 11912959fa6a4676a568298b7da1b71d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:43:09.580559 10809 raft_consensus.cc:482] T a1b9c1c4fbab46778c7cc36f8112f6fb P 11912959fa6a4676a568298b7da1b71d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:43:09.580698 10809 raft_consensus.cc:3037] T a1b9c1c4fbab46778c7cc36f8112f6fb P 11912959fa6a4676a568298b7da1b71d [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:43:09.582762 10809 raft_consensus.cc:504] T a1b9c1c4fbab46778c7cc36f8112f6fb P 11912959fa6a4676a568298b7da1b71d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11912959fa6a4676a568298b7da1b71d" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 37319 } } peers { permanent_uuid: "a3a2e72dd62d45e19ecba5c9b37fb8d8" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 38733 } } peers { permanent_uuid: "effdc8a6cb9a4b4daad6a2eef0ce7c42" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 39445 } }
I20260620 18:43:09.583307 10809 leader_election.cc:288] T a1b9c1c4fbab46778c7cc36f8112f6fb P 11912959fa6a4676a568298b7da1b71d [CANDIDATE]: Term 1 election: Requested vote from peers a3a2e72dd62d45e19ecba5c9b37fb8d8 (127.23.115.2:38733), effdc8a6cb9a4b4daad6a2eef0ce7c42 (127.23.115.1:39445)
I20260620 18:43:09.583702 10609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1b9c1c4fbab46778c7cc36f8112f6fb" candidate_uuid: "11912959fa6a4676a568298b7da1b71d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3a2e72dd62d45e19ecba5c9b37fb8d8"
I20260620 18:43:09.583762 10478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1b9c1c4fbab46778c7cc36f8112f6fb" candidate_uuid: "11912959fa6a4676a568298b7da1b71d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "effdc8a6cb9a4b4daad6a2eef0ce7c42"
I20260620 18:43:09.583947 10478 raft_consensus.cc:3037] T a1b9c1c4fbab46778c7cc36f8112f6fb P effdc8a6cb9a4b4daad6a2eef0ce7c42 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:43:09.583945 10609 raft_consensus.cc:3037] T a1b9c1c4fbab46778c7cc36f8112f6fb P a3a2e72dd62d45e19ecba5c9b37fb8d8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:43:09.587095 10609 raft_consensus.cc:2454] T a1b9c1c4fbab46778c7cc36f8112f6fb P a3a2e72dd62d45e19ecba5c9b37fb8d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 11912959fa6a4676a568298b7da1b71d in term 1.
I20260620 18:43:09.587095 10478 raft_consensus.cc:2454] T a1b9c1c4fbab46778c7cc36f8112f6fb P effdc8a6cb9a4b4daad6a2eef0ce7c42 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 11912959fa6a4676a568298b7da1b71d in term 1.
I20260620 18:43:09.587450 10675 leader_election.cc:302] T a1b9c1c4fbab46778c7cc36f8112f6fb P 11912959fa6a4676a568298b7da1b71d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11912959fa6a4676a568298b7da1b71d, a3a2e72dd62d45e19ecba5c9b37fb8d8; no voters:
I20260620 18:43:09.587702 10809 raft_consensus.cc:2781] T a1b9c1c4fbab46778c7cc36f8112f6fb P 11912959fa6a4676a568298b7da1b71d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:43:09.588105 10809 raft_consensus.cc:686] T a1b9c1c4fbab46778c7cc36f8112f6fb P 11912959fa6a4676a568298b7da1b71d [term 1 LEADER]: Becoming Leader. State: Replica: 11912959fa6a4676a568298b7da1b71d, State: Running, Role: LEADER
I20260620 18:43:09.588542 10809 consensus_queue.cc:237] T a1b9c1c4fbab46778c7cc36f8112f6fb P 11912959fa6a4676a568298b7da1b71d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11912959fa6a4676a568298b7da1b71d" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 37319 } } peers { permanent_uuid: "a3a2e72dd62d45e19ecba5c9b37fb8d8" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 38733 } } peers { permanent_uuid: "effdc8a6cb9a4b4daad6a2eef0ce7c42" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 39445 } }
I20260620 18:43:09.593358 10336 catalog_manager.cc:5495] T a1b9c1c4fbab46778c7cc36f8112f6fb P 11912959fa6a4676a568298b7da1b71d reported cstate change: term changed from 0 to 1, leader changed from <none> to 11912959fa6a4676a568298b7da1b71d (127.23.115.3). New cstate: current_term: 1 leader_uuid: "11912959fa6a4676a568298b7da1b71d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11912959fa6a4676a568298b7da1b71d" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 37319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3a2e72dd62d45e19ecba5c9b37fb8d8" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 38733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "effdc8a6cb9a4b4daad6a2eef0ce7c42" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 39445 } health_report { overall_health: UNKNOWN } } }
W20260620 18:43:09.616492 10785 tablet.cc:2369] T a1b9c1c4fbab46778c7cc36f8112f6fb P 11912959fa6a4676a568298b7da1b71d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:43:09.628705 10798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:09.623455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41506 (local address 127.23.115.3:37319)
0620 18:43:09.623644 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:09.623670 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:09.623690 (+ 20us) server_negotiation.cc:407] Connection header received
0620 18:43:09.623779 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:09.623783 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:09.623866 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:09.623987 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:43:09.624353 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:09.625052 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:09.626566 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:09.627018 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:09.627122 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:09.627669 (+ 547us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:43:09.628153 (+ 484us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:43:09.628303 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:09.628461 (+ 158us) server_negotiation.cc:299] Negotiation successful
0620 18:43:09.628504 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 18:43:09.638574 10478 raft_consensus.cc:1261] T a1b9c1c4fbab46778c7cc36f8112f6fb P effdc8a6cb9a4b4daad6a2eef0ce7c42 [term 1 FOLLOWER]: Refusing update from remote peer 11912959fa6a4676a568298b7da1b71d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:43:09.638572 10609 raft_consensus.cc:1261] T a1b9c1c4fbab46778c7cc36f8112f6fb P a3a2e72dd62d45e19ecba5c9b37fb8d8 [term 1 FOLLOWER]: Refusing update from remote peer 11912959fa6a4676a568298b7da1b71d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:43:09.639577 10809 consensus_queue.cc:1035] T a1b9c1c4fbab46778c7cc36f8112f6fb P 11912959fa6a4676a568298b7da1b71d [LEADER]: Connected to new peer: Peer: permanent_uuid: "effdc8a6cb9a4b4daad6a2eef0ce7c42" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 39445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:43:09.639884 10812 consensus_queue.cc:1035] T a1b9c1c4fbab46778c7cc36f8112f6fb P 11912959fa6a4676a568298b7da1b71d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3a2e72dd62d45e19ecba5c9b37fb8d8" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 38733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:43:09.656636 10817 mvcc.cc:205] Tried to move back new op lower bound from 7298994133547810816 to 7298994133358178304. Current Snapshot: MvccSnapshot[applied={T|T < 7298994133547810816}]
I20260620 18:43:09.667979 10336 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54640:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:43:09.668241 10336 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54640:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:43:09.672088 10336 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ed966b542890442288240d4737192b93: Sending DeleteTablet for 3 replicas of tablet a1b9c1c4fbab46778c7cc36f8112f6fb
I20260620 18:43:09.673465 10589 tablet_service.cc:1514] Processing DeleteTablet for tablet a1b9c1c4fbab46778c7cc36f8112f6fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:43:09 UTC) from {username='slave'} at 127.0.0.1:34504
I20260620 18:43:09.673805 10458 tablet_service.cc:1514] Processing DeleteTablet for tablet a1b9c1c4fbab46778c7cc36f8112f6fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:43:09 UTC) from {username='slave'} at 127.0.0.1:42122
I20260620 18:43:09.673933 10719 tablet_service.cc:1514] Processing DeleteTablet for tablet a1b9c1c4fbab46778c7cc36f8112f6fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:43:09 UTC) from {username='slave'} at 127.0.0.1:41500
I20260620 18:43:09.675145 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 10396
I20260620 18:43:09.683444 10826 tablet_replica.cc:330] T a1b9c1c4fbab46778c7cc36f8112f6fb P 11912959fa6a4676a568298b7da1b71d: stopping tablet replica
I20260620 18:43:09.683951 10826 raft_consensus.cc:2229] T a1b9c1c4fbab46778c7cc36f8112f6fb P 11912959fa6a4676a568298b7da1b71d [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:43:09.684991 10825 tablet_replica.cc:330] T a1b9c1c4fbab46778c7cc36f8112f6fb P effdc8a6cb9a4b4daad6a2eef0ce7c42: stopping tablet replica
I20260620 18:43:09.685429 10825 raft_consensus.cc:2229] T a1b9c1c4fbab46778c7cc36f8112f6fb P effdc8a6cb9a4b4daad6a2eef0ce7c42 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:43:09.685992 10825 raft_consensus.cc:2258] T a1b9c1c4fbab46778c7cc36f8112f6fb P effdc8a6cb9a4b4daad6a2eef0ce7c42 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:43:09.686720 10826 raft_consensus.cc:2258] T a1b9c1c4fbab46778c7cc36f8112f6fb P 11912959fa6a4676a568298b7da1b71d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:43:09.689384 10826 ts_tablet_manager.cc:1905] T a1b9c1c4fbab46778c7cc36f8112f6fb P 11912959fa6a4676a568298b7da1b71d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:43:09.689684 10824 tablet_replica.cc:330] T a1b9c1c4fbab46778c7cc36f8112f6fb P a3a2e72dd62d45e19ecba5c9b37fb8d8: stopping tablet replica
I20260620 18:43:09.690112 10824 raft_consensus.cc:2229] T a1b9c1c4fbab46778c7cc36f8112f6fb P a3a2e72dd62d45e19ecba5c9b37fb8d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:43:09.690639 10824 raft_consensus.cc:2258] T a1b9c1c4fbab46778c7cc36f8112f6fb P a3a2e72dd62d45e19ecba5c9b37fb8d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:43:09.689402 10825 ts_tablet_manager.cc:1905] T a1b9c1c4fbab46778c7cc36f8112f6fb P effdc8a6cb9a4b4daad6a2eef0ce7c42: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:43:09.696835 10801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:09.676560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42128 (local address 127.23.115.1:39445)
0620 18:43:09.676742 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:09.676748 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:09.687842 (+ 11094us) server_negotiation.cc:407] Connection header received
0620 18:43:09.688339 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:09.688344 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:09.688449 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:09.688536 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:09.689217 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:09.690307 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:09.692200 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:09.692846 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:09.692951 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:09.695097 (+ 2146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:09.695109 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:09.695126 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:43:09.695900 (+ 774us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:09.695969 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:43:09.695993 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:09.696086 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:09.696331 (+ 245us) server_negotiation.cc:299] Negotiation successful
0620 18:43:09.696454 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 18:43:09.697758 10826 ts_tablet_manager.cc:1918] T a1b9c1c4fbab46778c7cc36f8112f6fb P 11912959fa6a4676a568298b7da1b71d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:43:09.697902 10825 ts_tablet_manager.cc:1918] T a1b9c1c4fbab46778c7cc36f8112f6fb P effdc8a6cb9a4b4daad6a2eef0ce7c42: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:43:09.698045 10826 log.cc:1196] T a1b9c1c4fbab46778c7cc36f8112f6fb P 11912959fa6a4676a568298b7da1b71d: Deleting WAL directory at /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980676253068-24012-0/minicluster-data/ts-2/wal/wals/a1b9c1c4fbab46778c7cc36f8112f6fb
I20260620 18:43:09.698104 10825 log.cc:1196] T a1b9c1c4fbab46778c7cc36f8112f6fb P effdc8a6cb9a4b4daad6a2eef0ce7c42: Deleting WAL directory at /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980676253068-24012-0/minicluster-data/ts-0/wal/wals/a1b9c1c4fbab46778c7cc36f8112f6fb
I20260620 18:43:09.698163 10518 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:43:09.698712 10826 ts_tablet_manager.cc:1939] T a1b9c1c4fbab46778c7cc36f8112f6fb P 11912959fa6a4676a568298b7da1b71d: Deleting consensus metadata
I20260620 18:43:09.700001 10824 ts_tablet_manager.cc:1905] T a1b9c1c4fbab46778c7cc36f8112f6fb P a3a2e72dd62d45e19ecba5c9b37fb8d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:43:09.700832 10323 catalog_manager.cc:4841] TS 11912959fa6a4676a568298b7da1b71d (127.23.115.3:37319): tablet a1b9c1c4fbab46778c7cc36f8112f6fb (table test-table [id=369c6c2a27514010aa4a8b74dc55f543]) successfully deleted
I20260620 18:43:09.709009 10824 ts_tablet_manager.cc:1918] T a1b9c1c4fbab46778c7cc36f8112f6fb P a3a2e72dd62d45e19ecba5c9b37fb8d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:43:09.709211 10824 log.cc:1196] T a1b9c1c4fbab46778c7cc36f8112f6fb P a3a2e72dd62d45e19ecba5c9b37fb8d8: Deleting WAL directory at /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980676253068-24012-0/minicluster-data/ts-1/wal/wals/a1b9c1c4fbab46778c7cc36f8112f6fb
I20260620 18:43:09.709842 10824 ts_tablet_manager.cc:1939] T a1b9c1c4fbab46778c7cc36f8112f6fb P a3a2e72dd62d45e19ecba5c9b37fb8d8: Deleting consensus metadata
I20260620 18:43:09.711858 10324 catalog_manager.cc:4841] TS a3a2e72dd62d45e19ecba5c9b37fb8d8 (127.23.115.2:38733): tablet a1b9c1c4fbab46778c7cc36f8112f6fb (table test-table [id=369c6c2a27514010aa4a8b74dc55f543]) successfully deleted
I20260620 18:43:09.811720 10825 ts_tablet_manager.cc:1939] T a1b9c1c4fbab46778c7cc36f8112f6fb P effdc8a6cb9a4b4daad6a2eef0ce7c42: Deleting consensus metadata
I20260620 18:43:09.812073 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 10396
I20260620 18:43:09.826620 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 10527
W20260620 18:43:09.826776 10324 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.115.1:39445 (error 108)
I20260620 18:43:09.829506 10800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:09.828791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.1:39445 (local address 127.0.0.1:42134)
0620 18:43:09.828974 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:09.829104 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.1:39445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260620 18:43:09.829814 10324 catalog_manager.cc:4584] TS effdc8a6cb9a4b4daad6a2eef0ce7c42 (127.23.115.1:39445): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a1b9c1c4fbab46778c7cc36f8112f6fb: Network error: Client connection negotiation failed: client connection to 127.23.115.1:39445: connect: Connection refused (error 111)
I20260620 18:43:09.835642 10799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:09.827478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34506 (local address 127.23.115.2:38733)
0620 18:43:09.827659 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:09.827665 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:09.827687 (+ 22us) server_negotiation.cc:407] Connection header received
0620 18:43:09.827863 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:09.827869 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:09.828017 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:09.828128 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:09.828578 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:09.829850 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:09.831697 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:09.832115 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:09.832475 (+ 360us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:09.833830 (+ 1355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:09.833843 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:09.833862 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:43:09.834683 (+ 821us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:09.834757 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:43:09.834785 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:09.835095 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:09.835159 (+ 64us) server_negotiation.cc:299] Negotiation successful
0620 18:43:09.835311 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 18:43:09.836391 10649 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:43:09.883563 10800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:09.883025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.1:39445 (local address 127.0.0.1:42144)
0620 18:43:09.883241 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:09.883356 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.1:39445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260620 18:43:09.884680 10800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:09.884291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.1:39445 (local address 127.0.0.1:42158)
0620 18:43:09.884446 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:09.884528 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.1:39445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 18:43:09.936036 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 10527
I20260620 18:43:09.945133 10800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:09.944682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.1:39445 (local address 127.0.0.1:42160)
0620 18:43:09.944898 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:09.944984 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.1:39445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260620 18:43:09.946149 10800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:09.945839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.1:39445 (local address 127.0.0.1:42168)
0620 18:43:09.945980 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:09.946046 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.1:39445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260620 18:43:09.957887 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 10657
I20260620 18:43:09.966333 10798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:09.958783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41510 (local address 127.23.115.3:37319)
0620 18:43:09.958987 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:09.959094 (+ 107us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:09.959121 (+ 27us) server_negotiation.cc:407] Connection header received
0620 18:43:09.959245 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:09.959250 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:09.959335 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:09.959429 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:09.959910 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:09.960705 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:09.962525 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:09.963146 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:09.963256 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:09.964800 (+ 1544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:09.964808 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:09.964823 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:43:09.965515 (+ 692us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:09.965563 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:43:09.965585 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:09.965671 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:09.965896 (+ 225us) server_negotiation.cc:299] Negotiation successful
0620 18:43:09.965984 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 18:43:09.967132 10779 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:43:10.027789 10800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:10.027152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.1:39445 (local address 127.0.0.1:42170)
0620 18:43:10.027424 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:10.027558 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.1:39445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260620 18:43:10.029340 10800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:10.028887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.1:39445 (local address 127.0.0.1:42182)
0620 18:43:10.029112 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:10.029201 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.1:39445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260620 18:43:10.094048 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 10657
I20260620 18:43:10.109970 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 10306
I20260620 18:43:10.111016 10367 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:43:10.220849 10800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:10.220135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.1:39445 (local address 127.0.0.1:42188)
0620 18:43:10.220532 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:10.220639 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.1:39445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205}
I20260620 18:43:10.220893 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 10306
2026-06-20T18:43:10Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2794 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980676253068-24012-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:43:10 dist-test-slave-d645 krb5kdc[10835](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:43:10 dist-test-slave-d645 krb5kdc[10835](info): set up 2 sockets
Jun 20 18:43:10 dist-test-slave-d645 krb5kdc[10835](info): commencing operation
krb5kdc: starting...
W20260620 18:43:11.263341 24012 mini_kdc.cc:121] Time spent starting KDC: real 1.003s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980676253068-24012-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980676253068-24012-0/krb5kdc/test-user.keytab.
Jun 20 18:43:11 dist-test-slave-d645 krb5kdc[10835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980991, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:43:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:43:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.115.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.115.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.115.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab.
Entry for principal kudu/127.23.115.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab.
I20260620 18:43:11.508020 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980676253068-24012-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980676253068-24012-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980676253068-24012-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.115.62:43451
--webserver_interface=127.23.115.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab
--principal=kudu/127.23.115.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:44635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.115.62:43451
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:43:11.825563 10848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:43:11.825874 10848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:43:11.826010 10848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:43:11.833117 10848 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:43:11.833235 10848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:43:11.833312 10848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:43:11.833388 10848 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:43:11.833447 10848 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:43:11.843111 10848 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:44635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980676253068-24012-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980676253068-24012-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.115.62:43451
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab
--principal=kudu/127.23.115.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.115.62:43451
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.115.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980676253068-24012-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:43:11.844363 10848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:43:11.845592 10848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:43:11.856299 10856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:11.856426 10854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:11.856818 10853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:11.857508 10848 server_base.cc:1034] running on GCE node
I20260620 18:43:11.858309 10848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:43:11.859763 10848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:43:11.861011 10848 hybrid_clock.cc:648] HybridClock initialized: now 1781980991860956 us; error 41 us; skew 500 ppm
I20260620 18:43:11.861385 10848 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:43:11 dist-test-slave-d645 krb5kdc[10835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980991, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:43:11.868494 10848 init.cc:377] Logged in from keytab as kudu/127.23.115.62@KRBTEST.COM (short username kudu)
I20260620 18:43:11.870020 10848 webserver.cc:460] Webserver started at http://127.23.115.62:40569/ using document root <none> and password file <none>
I20260620 18:43:11.870684 10848 fs_manager.cc:359] Metadata directory not provided
I20260620 18:43:11.870800 10848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:43:11.871114 10848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:43:11.874028 10848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980676253068-24012-0/minicluster-data/master-0/data/instance:
uuid: "285063a052d2448da228e29fd3e3bd1e"
format_stamp: "Formatted at 2026-06-20 18:43:11 on dist-test-slave-d645"
server_key: "870b6cdd4ddaf714fcd44c4aa8f225c4"
server_key_iv: "c151b950ea4abd822f0c1316777b48bd"
server_key_version: "encryptionkey@0"
I20260620 18:43:11.874737 10848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980676253068-24012-0/minicluster-data/master-0/wal/instance:
uuid: "285063a052d2448da228e29fd3e3bd1e"
format_stamp: "Formatted at 2026-06-20 18:43:11 on dist-test-slave-d645"
server_key: "870b6cdd4ddaf714fcd44c4aa8f225c4"
server_key_iv: "c151b950ea4abd822f0c1316777b48bd"
server_key_version: "encryptionkey@0"
I20260620 18:43:11.879679 10848 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 18:43:11.883082 10863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:11.884377 10848 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 18:43:11.884538 10848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980676253068-24012-0/minicluster-data/master-0/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980676253068-24012-0/minicluster-data/master-0/wal
uuid: "285063a052d2448da228e29fd3e3bd1e"
format_stamp: "Formatted at 2026-06-20 18:43:11 on dist-test-slave-d645"
server_key: "870b6cdd4ddaf714fcd44c4aa8f225c4"
server_key_iv: "c151b950ea4abd822f0c1316777b48bd"
server_key_version: "encryptionkey@0"
I20260620 18:43:11.884716 10848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980676253068-24012-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980676253068-24012-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980676253068-24012-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:43:11.902954 10848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:43:11.907037 10848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:43:11.907297 10848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:43:11.925388 10848 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.62:43451
I20260620 18:43:11.925441 10914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.62:43451 every 8 connection(s)
I20260620 18:43:11.927230 10848 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
I20260620 18:43:11.931051 10915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:43:11.931174 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 10848
I20260620 18:43:11.931372 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980676253068-24012-0/minicluster-data/master-0/wal/instance
I20260620 18:43:11.931784 24012 external_mini_cluster.cc:1428] Setting key ad2146f767f0dd3ed6fe666082d80fee
I20260620 18:43:11.944975 10915 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 285063a052d2448da228e29fd3e3bd1e: Bootstrap starting.
Jun 20 18:43:11 dist-test-slave-d645 krb5kdc[10835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980991, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.115.62@KRBTEST.COM
I20260620 18:43:11.948051 10915 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 285063a052d2448da228e29fd3e3bd1e: Neither blocks nor log segments found. Creating new log.
I20260620 18:43:11.949344 10915 log.cc:824] T 00000000000000000000000000000000 P 285063a052d2448da228e29fd3e3bd1e: Log is configured to *not* fsync() on all Append() calls
I20260620 18:43:11.952574 10915 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 285063a052d2448da228e29fd3e3bd1e: No bootstrap required, opened a new log
I20260620 18:43:11.955999 10918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:11.933674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48666 (local address 127.23.115.62:43451)
0620 18:43:11.934479 (+ 805us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:11.934494 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:11.934543 (+ 49us) server_negotiation.cc:407] Connection header received
0620 18:43:11.937299 (+ 2756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:11.937337 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:11.937865 (+ 528us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:11.938335 (+ 470us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:11.939882 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:11.941829 (+ 1947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:11.943300 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:11.944277 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:11.944467 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:11.948298 (+ 3831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:11.948311 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:11.948345 (+ 34us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:11.948398 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:11.952091 (+ 3693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:11.952788 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:11.952794 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:11.952800 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:11.952985 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:11.953354 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:11.953358 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:11.953360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:11.953867 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:11.954167 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:11.954583 (+ 416us) server_negotiation.cc:299] Negotiation successful
0620 18:43:11.955038 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":329,"threads_started":1}
I20260620 18:43:11.960677 10915 raft_consensus.cc:348] T 00000000000000000000000000000000 P 285063a052d2448da228e29fd3e3bd1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "285063a052d2448da228e29fd3e3bd1e" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 43451 } }
I20260620 18:43:11.961017 10915 raft_consensus.cc:374] T 00000000000000000000000000000000 P 285063a052d2448da228e29fd3e3bd1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:43:11.961103 10915 raft_consensus.cc:729] T 00000000000000000000000000000000 P 285063a052d2448da228e29fd3e3bd1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 285063a052d2448da228e29fd3e3bd1e, State: Initialized, Role: FOLLOWER
I20260620 18:43:11.961604 10915 consensus_queue.cc:260] T 00000000000000000000000000000000 P 285063a052d2448da228e29fd3e3bd1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "285063a052d2448da228e29fd3e3bd1e" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 43451 } }
I20260620 18:43:11.961818 10915 raft_consensus.cc:388] T 00000000000000000000000000000000 P 285063a052d2448da228e29fd3e3bd1e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:43:11.961947 10915 raft_consensus.cc:482] T 00000000000000000000000000000000 P 285063a052d2448da228e29fd3e3bd1e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:43:11.962090 10915 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 285063a052d2448da228e29fd3e3bd1e [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:43:11.964355 10915 raft_consensus.cc:504] T 00000000000000000000000000000000 P 285063a052d2448da228e29fd3e3bd1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "285063a052d2448da228e29fd3e3bd1e" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 43451 } }
I20260620 18:43:11.964788 10915 leader_election.cc:302] T 00000000000000000000000000000000 P 285063a052d2448da228e29fd3e3bd1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 285063a052d2448da228e29fd3e3bd1e; no voters:
I20260620 18:43:11.965350 10915 leader_election.cc:288] T 00000000000000000000000000000000 P 285063a052d2448da228e29fd3e3bd1e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:43:11.965482 10920 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 285063a052d2448da228e29fd3e3bd1e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:43:11.966090 10920 raft_consensus.cc:686] T 00000000000000000000000000000000 P 285063a052d2448da228e29fd3e3bd1e [term 1 LEADER]: Becoming Leader. State: Replica: 285063a052d2448da228e29fd3e3bd1e, State: Running, Role: LEADER
I20260620 18:43:11.966645 10920 consensus_queue.cc:237] T 00000000000000000000000000000000 P 285063a052d2448da228e29fd3e3bd1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "285063a052d2448da228e29fd3e3bd1e" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 43451 } }
I20260620 18:43:11.967217 10915 sys_catalog.cc:564] T 00000000000000000000000000000000 P 285063a052d2448da228e29fd3e3bd1e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:43:11.969084 10922 sys_catalog.cc:455] T 00000000000000000000000000000000 P 285063a052d2448da228e29fd3e3bd1e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 285063a052d2448da228e29fd3e3bd1e. Latest consensus state: current_term: 1 leader_uuid: "285063a052d2448da228e29fd3e3bd1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "285063a052d2448da228e29fd3e3bd1e" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 43451 } } }
I20260620 18:43:11.969460 10922 sys_catalog.cc:458] T 00000000000000000000000000000000 P 285063a052d2448da228e29fd3e3bd1e [sys.catalog]: This master's current role is: LEADER
I20260620 18:43:11.969331 10921 sys_catalog.cc:455] T 00000000000000000000000000000000 P 285063a052d2448da228e29fd3e3bd1e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "285063a052d2448da228e29fd3e3bd1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "285063a052d2448da228e29fd3e3bd1e" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 43451 } } }
I20260620 18:43:11.969575 10921 sys_catalog.cc:458] T 00000000000000000000000000000000 P 285063a052d2448da228e29fd3e3bd1e [sys.catalog]: This master's current role is: LEADER
I20260620 18:43:11.971726 10928 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:43:11.976307 10928 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:43:11.984891 10928 catalog_manager.cc:1347] Generated new cluster ID: 3840e3a0a06e4b2298e7b6d50fbbf671
I20260620 18:43:11.985003 10928 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:43:12.003717 10928 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:43:12.005532 10928 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:43:12.022358 10928 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 285063a052d2448da228e29fd3e3bd1e: Generated new TSK 0
I20260620 18:43:12.023554 10928 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.115.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.115.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.115.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.1.keytab.
Entry for principal kudu/127.23.115.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.1.keytab.
I20260620 18:43:12.091441 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980676253068-24012-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980676253068-24012-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980676253068-24012-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980676253068-24012-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.115.1:0
--local_ip_for_outbound_sockets=127.23.115.1
--webserver_interface=127.23.115.1
--webserver_port=0
--tserver_master_addrs=127.23.115.62:43451
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.1.keytab
--principal=kudu/127.23.115.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:44635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:43:12.410774 10941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:43:12.411114 10941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:43:12.411262 10941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:43:12.418401 10941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:43:12.418498 10941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:43:12.418620 10941 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.115.1
I20260620 18:43:12.428073 10941 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:44635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980676253068-24012-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980676253068-24012-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.1.keytab
--principal=kudu/127.23.115.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.115.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980676253068-24012-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.115.1
--webserver_port=0
--tserver_master_addrs=127.23.115.62:43451
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.115.1
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980676253068-24012-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:43:12.429207 10941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:43:12.430393 10941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:43:12.442487 10946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:12.442479 10949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:12.442535 10947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:12.443212 10941 server_base.cc:1034] running on GCE node
I20260620 18:43:12.443819 10941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:43:12.444708 10941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:43:12.445917 10941 hybrid_clock.cc:648] HybridClock initialized: now 1781980992445868 us; error 38 us; skew 500 ppm
I20260620 18:43:12.446202 10941 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:43:12 dist-test-slave-d645 krb5kdc[10835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980992, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:43:12.449193 10941 init.cc:377] Logged in from keytab as kudu/127.23.115.1@KRBTEST.COM (short username kudu)
I20260620 18:43:12.450610 10941 webserver.cc:460] Webserver started at http://127.23.115.1:36615/ using document root <none> and password file <none>
I20260620 18:43:12.451321 10941 fs_manager.cc:359] Metadata directory not provided
I20260620 18:43:12.451424 10941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:43:12.451709 10941 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:43:12.454546 10941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980676253068-24012-0/minicluster-data/ts-0/data/instance:
uuid: "4b21af82df0c496eb1af09e4a6dac4cf"
format_stamp: "Formatted at 2026-06-20 18:43:12 on dist-test-slave-d645"
server_key: "2f93f390c9cde19066e08c79b167dc50"
server_key_iv: "10e23bf693363378a82f3ea338b4e648"
server_key_version: "encryptionkey@0"
I20260620 18:43:12.455281 10941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980676253068-24012-0/minicluster-data/ts-0/wal/instance:
uuid: "4b21af82df0c496eb1af09e4a6dac4cf"
format_stamp: "Formatted at 2026-06-20 18:43:12 on dist-test-slave-d645"
server_key: "2f93f390c9cde19066e08c79b167dc50"
server_key_iv: "10e23bf693363378a82f3ea338b4e648"
server_key_version: "encryptionkey@0"
I20260620 18:43:12.460071 10941 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 18:43:12.463356 10956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:12.464879 10941 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 18:43:12.465045 10941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980676253068-24012-0/minicluster-data/ts-0/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980676253068-24012-0/minicluster-data/ts-0/wal
uuid: "4b21af82df0c496eb1af09e4a6dac4cf"
format_stamp: "Formatted at 2026-06-20 18:43:12 on dist-test-slave-d645"
server_key: "2f93f390c9cde19066e08c79b167dc50"
server_key_iv: "10e23bf693363378a82f3ea338b4e648"
server_key_version: "encryptionkey@0"
I20260620 18:43:12.465243 10941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980676253068-24012-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980676253068-24012-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980676253068-24012-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:43:12.490389 10941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:43:12.494328 10941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:43:12.494581 10941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:43:12.495553 10941 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:43:12.496841 10941 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:43:12.496917 10941 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:12.497031 10941 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:43:12.497081 10941 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:12.527801 10941 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.1:46291
I20260620 18:43:12.527868 11068 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.1:46291 every 8 connection(s)
I20260620 18:43:12.529026 10941 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980676253068-24012-0/minicluster-data/ts-0/data/info.pb
I20260620 18:43:12.537667 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 10941
I20260620 18:43:12.537869 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980676253068-24012-0/minicluster-data/ts-0/wal/instance
I20260620 18:43:12.538395 24012 external_mini_cluster.cc:1428] Setting key 05b9d9bae3e7cbba4ccaa6539b4df67a
Jun 20 18:43:12 dist-test-slave-d645 krb5kdc[10835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980992, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.1@KRBTEST.COM for kudu/127.23.115.62@KRBTEST.COM
I20260620 18:43:12.550376 11071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:12.531560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.1:47637 (local address 127.23.115.62:43451)
0620 18:43:12.531978 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:12.531983 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:12.532743 (+ 760us) server_negotiation.cc:407] Connection header received
0620 18:43:12.534563 (+ 1820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:12.534569 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:12.534655 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:12.534747 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:12.536372 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:12.537083 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:12.539242 (+ 2159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:12.539649 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:12.539774 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:12.543840 (+ 4066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:12.543846 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:12.543849 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:12.543885 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:12.546316 (+ 2431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:12.547224 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:12.547228 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:12.547231 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:12.547381 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:12.548022 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:12.548034 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:12.548037 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:12.548667 (+ 630us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:12.548808 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:12.549727 (+ 919us) server_negotiation.cc:299] Negotiation successful
0620 18:43:12.549963 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":210,"threads_started":1}
I20260620 18:43:12.551113 11072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:12.531939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:43451 (local address 127.23.115.1:47637)
0620 18:43:12.532596 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:12.532637 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:12.534309 (+ 1672us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:12.535031 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:12.535042 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:12.535642 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:43:12.536184 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:12.536200 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:12.537235 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:12.537238 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:12.539108 (+ 1870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:12.539121 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:12.539864 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:12.539871 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:12.540109 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:12.540800 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:43:12.540832 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:43:12.543587 (+ 2755us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:43:12.546458 (+ 2871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:12.546467 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:12.546485 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:12.547075 (+ 590us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:12.547486 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:12.547490 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:12.547494 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:12.547745 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:12.548826 (+ 1081us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:43:12.548835 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:43:12.549295 (+ 460us) client_negotiation.cc:769] Sending connection context
0620 18:43:12.549562 (+ 267us) client_negotiation.cc:240] Negotiation successful
0620 18:43:12.549979 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":257,"threads_started":1}
I20260620 18:43:12.552642 11069 heartbeater.cc:347] Connected to a master server at 127.23.115.62:43451
I20260620 18:43:12.552934 11069 heartbeater.cc:464] Registering TS with master...
I20260620 18:43:12.553602 11069 heartbeater.cc:511] Master 127.23.115.62:43451 requested a full tablet report, sending...
I20260620 18:43:12.555575 10880 ts_manager.cc:194] Registered new tserver with Master: 4b21af82df0c496eb1af09e4a6dac4cf (127.23.115.1:46291)
I20260620 18:43:12.556964 10880 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.115.1@KRBTEST.COM'} at 127.23.115.1:47637
WARNING: no policy specified for kudu/127.23.115.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.115.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.115.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.2.keytab.
Entry for principal kudu/127.23.115.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.2.keytab.
I20260620 18:43:12.590221 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980676253068-24012-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980676253068-24012-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980676253068-24012-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980676253068-24012-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.115.2:0
--local_ip_for_outbound_sockets=127.23.115.2
--webserver_interface=127.23.115.2
--webserver_port=0
--tserver_master_addrs=127.23.115.62:43451
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.2.keytab
--principal=kudu/127.23.115.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:44635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:43:12.904059 11075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:43:12.904340 11075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:43:12.904484 11075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:43:12.911595 11075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:43:12.911722 11075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:43:12.911881 11075 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.115.2
I20260620 18:43:12.921466 11075 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:44635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980676253068-24012-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980676253068-24012-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.2.keytab
--principal=kudu/127.23.115.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.115.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980676253068-24012-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.115.2
--webserver_port=0
--tserver_master_addrs=127.23.115.62:43451
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.115.2
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980676253068-24012-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:43:12.922722 11075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:43:12.924021 11075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:43:12.938313 11083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:12.938313 11081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:12.938851 11080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:12.939225 11075 server_base.cc:1034] running on GCE node
I20260620 18:43:12.939868 11075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:43:12.940922 11075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:43:12.942170 11075 hybrid_clock.cc:648] HybridClock initialized: now 1781980992942120 us; error 34 us; skew 500 ppm
I20260620 18:43:12.942505 11075 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:43:12 dist-test-slave-d645 krb5kdc[10835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980992, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:43:12.945574 11075 init.cc:377] Logged in from keytab as kudu/127.23.115.2@KRBTEST.COM (short username kudu)
I20260620 18:43:12.947021 11075 webserver.cc:460] Webserver started at http://127.23.115.2:38355/ using document root <none> and password file <none>
I20260620 18:43:12.947631 11075 fs_manager.cc:359] Metadata directory not provided
I20260620 18:43:12.947729 11075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:43:12.947983 11075 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:43:12.950693 11075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980676253068-24012-0/minicluster-data/ts-1/data/instance:
uuid: "319393d00cc24530b55827f1521cdc36"
format_stamp: "Formatted at 2026-06-20 18:43:12 on dist-test-slave-d645"
server_key: "3246385b3dd435e4b31e8d1fe000205d"
server_key_iv: "f6831c1b4badaabe20ddaac4a0170c6a"
server_key_version: "encryptionkey@0"
I20260620 18:43:12.951509 11075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980676253068-24012-0/minicluster-data/ts-1/wal/instance:
uuid: "319393d00cc24530b55827f1521cdc36"
format_stamp: "Formatted at 2026-06-20 18:43:12 on dist-test-slave-d645"
server_key: "3246385b3dd435e4b31e8d1fe000205d"
server_key_iv: "f6831c1b4badaabe20ddaac4a0170c6a"
server_key_version: "encryptionkey@0"
I20260620 18:43:12.956993 11075 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260620 18:43:12.960278 11090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:12.961489 11075 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:43:12.961621 11075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980676253068-24012-0/minicluster-data/ts-1/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980676253068-24012-0/minicluster-data/ts-1/wal
uuid: "319393d00cc24530b55827f1521cdc36"
format_stamp: "Formatted at 2026-06-20 18:43:12 on dist-test-slave-d645"
server_key: "3246385b3dd435e4b31e8d1fe000205d"
server_key_iv: "f6831c1b4badaabe20ddaac4a0170c6a"
server_key_version: "encryptionkey@0"
I20260620 18:43:12.961819 11075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980676253068-24012-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980676253068-24012-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980676253068-24012-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:43:12.979627 11075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:43:12.984035 11075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:43:12.984284 11075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:43:12.985229 11075 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:43:12.986567 11075 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:43:12.986639 11075 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:12.986734 11075 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:43:12.986778 11075 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:13.018836 11202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.2:33641 every 8 connection(s)
I20260620 18:43:13.019078 11075 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.2:33641
I20260620 18:43:13.020803 11075 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980676253068-24012-0/minicluster-data/ts-1/data/info.pb
I20260620 18:43:13.022794 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 11075
I20260620 18:43:13.022989 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980676253068-24012-0/minicluster-data/ts-1/wal/instance
I20260620 18:43:13.023527 24012 external_mini_cluster.cc:1428] Setting key 186c127117fe1fce9934a735ca2a0a77
Jun 20 18:43:13 dist-test-slave-d645 krb5kdc[10835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980992, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.2@KRBTEST.COM for kudu/127.23.115.62@KRBTEST.COM
I20260620 18:43:13.046103 11071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:13.024791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.2:41929 (local address 127.23.115.62:43451)
0620 18:43:13.024978 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:13.024984 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:13.026323 (+ 1339us) server_negotiation.cc:407] Connection header received
0620 18:43:13.028836 (+ 2513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:13.028841 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:13.028912 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:13.029001 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:13.031002 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:13.032049 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:13.034275 (+ 2226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:13.034781 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:13.034900 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:13.039201 (+ 4301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:13.039208 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:13.039212 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:13.039251 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:13.042235 (+ 2984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:13.043008 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:13.043013 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:13.043016 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:13.043159 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:13.043951 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:13.043955 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:13.043958 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:13.044292 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:13.044438 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:13.045520 (+ 1082us) server_negotiation.cc:299] Negotiation successful
0620 18:43:13.045766 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 18:43:13.047072 11205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:13.025328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:43451 (local address 127.23.115.2:41929)
0620 18:43:13.026149 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:13.026204 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:13.028503 (+ 2299us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:13.029261 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:13.029271 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:13.029944 (+ 673us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:43:13.030781 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:13.030804 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:13.032275 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:13.032280 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:13.034090 (+ 1810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:13.034105 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:13.035046 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:13.035053 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:13.035360 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:13.036180 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:43:13.036222 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:43:13.038953 (+ 2731us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:43:13.042389 (+ 3436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:13.042397 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:13.042411 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:13.042866 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:13.043423 (+ 557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:13.043429 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:13.043432 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:13.043784 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:13.044481 (+ 697us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:43:13.044489 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:43:13.045115 (+ 626us) client_negotiation.cc:769] Sending connection context
0620 18:43:13.045404 (+ 289us) client_negotiation.cc:240] Negotiation successful
0620 18:43:13.045843 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":326,"threads_started":1}
I20260620 18:43:13.048659 11203 heartbeater.cc:347] Connected to a master server at 127.23.115.62:43451
I20260620 18:43:13.048954 11203 heartbeater.cc:464] Registering TS with master...
I20260620 18:43:13.049674 11203 heartbeater.cc:511] Master 127.23.115.62:43451 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.23.115.3@KRBTEST.COM; defaulting to no policy
I20260620 18:43:13.051174 10880 ts_manager.cc:194] Registered new tserver with Master: 319393d00cc24530b55827f1521cdc36 (127.23.115.2:33641)
I20260620 18:43:13.052060 10880 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.115.2@KRBTEST.COM'} at 127.23.115.2:41929
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.115.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.115.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.3.keytab.
Entry for principal kudu/127.23.115.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.3.keytab.
I20260620 18:43:13.080227 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980676253068-24012-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980676253068-24012-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980676253068-24012-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980676253068-24012-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.115.3:0
--local_ip_for_outbound_sockets=127.23.115.3
--webserver_interface=127.23.115.3
--webserver_port=0
--tserver_master_addrs=127.23.115.62:43451
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.3.keytab
--principal=kudu/127.23.115.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:44635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:43:13.396607 11208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:43:13.396872 11208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:43:13.396965 11208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:43:13.404059 11208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:43:13.404157 11208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:43:13.404273 11208 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.115.3
I20260620 18:43:13.413679 11208 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:44635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980676253068-24012-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980676253068-24012-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.3.keytab
--principal=kudu/127.23.115.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.115.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980676253068-24012-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.115.3
--webserver_port=0
--tserver_master_addrs=127.23.115.62:43451
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.115.3
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980676253068-24012-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:43:13.414840 11208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:43:13.416049 11208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:43:13.428319 11216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:13.428345 11214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:13.428740 11213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:13.429322 11208 server_base.cc:1034] running on GCE node
I20260620 18:43:13.429921 11208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:43:13.430845 11208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:43:13.432374 11208 hybrid_clock.cc:648] HybridClock initialized: now 1781980993432308 us; error 48 us; skew 500 ppm
I20260620 18:43:13.432713 11208 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:43:13 dist-test-slave-d645 krb5kdc[10835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980993, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:43:13.435745 11208 init.cc:377] Logged in from keytab as kudu/127.23.115.3@KRBTEST.COM (short username kudu)
I20260620 18:43:13.437162 11208 webserver.cc:460] Webserver started at http://127.23.115.3:37311/ using document root <none> and password file <none>
I20260620 18:43:13.437762 11208 fs_manager.cc:359] Metadata directory not provided
I20260620 18:43:13.437865 11208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:43:13.438110 11208 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:43:13.441039 11208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980676253068-24012-0/minicluster-data/ts-2/data/instance:
uuid: "e32dd350c9af47c2a8a461e504f9188a"
format_stamp: "Formatted at 2026-06-20 18:43:13 on dist-test-slave-d645"
server_key: "69133a30a5cd94fd5e3d19fb923caaeb"
server_key_iv: "9939605f2e845727fab832e0efeaa744"
server_key_version: "encryptionkey@0"
I20260620 18:43:13.441749 11208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980676253068-24012-0/minicluster-data/ts-2/wal/instance:
uuid: "e32dd350c9af47c2a8a461e504f9188a"
format_stamp: "Formatted at 2026-06-20 18:43:13 on dist-test-slave-d645"
server_key: "69133a30a5cd94fd5e3d19fb923caaeb"
server_key_iv: "9939605f2e845727fab832e0efeaa744"
server_key_version: "encryptionkey@0"
I20260620 18:43:13.446746 11208 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 18:43:13.450161 11223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:13.451622 11208 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:43:13.451784 11208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980676253068-24012-0/minicluster-data/ts-2/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980676253068-24012-0/minicluster-data/ts-2/wal
uuid: "e32dd350c9af47c2a8a461e504f9188a"
format_stamp: "Formatted at 2026-06-20 18:43:13 on dist-test-slave-d645"
server_key: "69133a30a5cd94fd5e3d19fb923caaeb"
server_key_iv: "9939605f2e845727fab832e0efeaa744"
server_key_version: "encryptionkey@0"
I20260620 18:43:13.451987 11208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980676253068-24012-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980676253068-24012-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980676253068-24012-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:43:13.468657 11208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:43:13.472766 11208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:43:13.473029 11208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:43:13.473951 11208 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:43:13.475325 11208 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:43:13.475400 11208 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:13.475518 11208 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:43:13.475570 11208 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:13.505668 11208 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.3:42473
I20260620 18:43:13.505713 11335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.3:42473 every 8 connection(s)
I20260620 18:43:13.506865 11208 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980676253068-24012-0/minicluster-data/ts-2/data/info.pb
I20260620 18:43:13.513487 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 11208
I20260620 18:43:13.513700 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980676253068-24012-0/minicluster-data/ts-2/wal/instance
I20260620 18:43:13.514204 24012 external_mini_cluster.cc:1428] Setting key 4339101a8fe7bed7741733d1b81680c1
Jun 20 18:43:13 dist-test-slave-d645 krb5kdc[10835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980993, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.3@KRBTEST.COM for kudu/127.23.115.62@KRBTEST.COM
I20260620 18:43:13.529776 11071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:13.509468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.3:58573 (local address 127.23.115.62:43451)
0620 18:43:13.509787 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:13.509792 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:13.510622 (+ 830us) server_negotiation.cc:407] Connection header received
0620 18:43:13.512417 (+ 1795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:13.512422 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:13.512503 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:13.512595 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:13.514473 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:13.515498 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:13.517391 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:13.517906 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:13.518115 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:13.522435 (+ 4320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:13.522440 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:13.522443 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:13.522476 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:13.525223 (+ 2747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:13.526054 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:13.526059 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:13.526063 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:13.526195 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:13.526896 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:13.526921 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:13.526923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:13.527237 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:13.527384 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:13.528804 (+ 1420us) server_negotiation.cc:299] Negotiation successful
0620 18:43:13.529156 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145}
I20260620 18:43:13.530131 11338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:13.509736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:43451 (local address 127.23.115.3:58573)
0620 18:43:13.510453 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:13.510488 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:13.512145 (+ 1657us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:13.512822 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:13.512830 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:13.513520 (+ 690us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:43:13.514292 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:13.514318 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:13.515672 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:13.515678 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:13.517177 (+ 1499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:13.517190 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:13.518308 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:13.518316 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:13.518657 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:13.519511 (+ 854us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:43:13.519537 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:43:13.522165 (+ 2628us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:43:13.525392 (+ 3227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:13.525400 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:13.525413 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:13.525905 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:13.526384 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:13.526389 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:13.526392 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:13.526745 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:13.527397 (+ 652us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:43:13.527406 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:43:13.527845 (+ 439us) client_negotiation.cc:769] Sending connection context
0620 18:43:13.528105 (+ 260us) client_negotiation.cc:240] Negotiation successful
0620 18:43:13.528569 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":287,"threads_started":1}
I20260620 18:43:13.531646 11336 heartbeater.cc:347] Connected to a master server at 127.23.115.62:43451
I20260620 18:43:13.531985 11336 heartbeater.cc:464] Registering TS with master...
I20260620 18:43:13.532801 11336 heartbeater.cc:511] Master 127.23.115.62:43451 requested a full tablet report, sending...
I20260620 18:43:13.534448 10880 ts_manager.cc:194] Registered new tserver with Master: e32dd350c9af47c2a8a461e504f9188a (127.23.115.3:42473)
I20260620 18:43:13.535320 10880 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.115.3@KRBTEST.COM'} at 127.23.115.3:58573
I20260620 18:43:13.542448 24012 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:43:13.559749 11071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:13.546497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48680 (local address 127.23.115.62:43451)
0620 18:43:13.546752 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:13.546760 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:13.547130 (+ 370us) server_negotiation.cc:407] Connection header received
0620 18:43:13.547346 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:13.547350 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:13.547415 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:13.547537 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:13.549036 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:13.549785 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:13.551202 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:13.551662 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:13.551820 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:13.554229 (+ 2409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:13.554237 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:13.554242 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:13.554283 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:13.556970 (+ 2687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:13.557642 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:13.557648 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:13.557652 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:13.557783 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:13.558152 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:13.558158 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:13.558162 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:13.558540 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:13.558786 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:13.559149 (+ 363us) server_negotiation.cc:299] Negotiation successful
0620 18:43:13.559361 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 18:43:13.560003 11069 heartbeater.cc:503] Master 127.23.115.62:43451 was elected leader, sending a full tablet report...
I20260620 18:43:13.563635 10879 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48680:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:43:13.570509 10879 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:43:13.603174 11350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:13.582369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39772 (local address 127.23.115.2:33641)
0620 18:43:13.594332 (+ 11963us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:13.594341 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:13.594372 (+ 31us) server_negotiation.cc:407] Connection header received
0620 18:43:13.594484 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:13.594493 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:13.594742 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:13.595693 (+ 951us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:43:13.595926 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:13.597525 (+ 1599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:13.600374 (+ 2849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:13.601902 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:13.602031 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:13.602210 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:13.602661 (+ 451us) server_negotiation.cc:299] Negotiation successful
0620 18:43:13.602822 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11803,"thread_start_us":11669,"threads_started":1}
I20260620 18:43:13.603719 11349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:13.582245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.2:33641 (local address 127.0.0.1:39772)
0620 18:43:13.593514 (+ 11269us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:13.593561 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:13.593727 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:13.595031 (+ 1304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:13.595039 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:13.595073 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:43:13.595532 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:13.595544 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:13.597696 (+ 2152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:13.597702 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:13.600196 (+ 2494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:13.600208 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:13.602308 (+ 2100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:13.602312 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:13.602433 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:13.602473 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 18:43:13.602573 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 18:43:13.602745 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11033,"thread_start_us":14274,"threads_started":1}
I20260620 18:43:13.605332 11348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:13.582294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.1:46291 (local address 127.0.0.1:34536)
0620 18:43:13.596370 (+ 14076us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:13.596390 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:13.597453 (+ 1063us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:13.598377 (+ 924us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:13.598381 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:13.598406 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:43:13.598847 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:13.598872 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:13.600929 (+ 2057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:13.600934 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:13.602969 (+ 2035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:13.602982 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:13.604698 (+ 1716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:13.604704 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:13.604875 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:13.604906 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 18:43:13.604989 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 18:43:13.605110 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13945,"thread_start_us":10855,"threads_started":1}
I20260620 18:43:13.605536 11351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:13.582474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34536 (local address 127.23.115.1:46291)
0620 18:43:13.592837 (+ 10363us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:13.592847 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:13.596498 (+ 3651us) server_negotiation.cc:407] Connection header received
0620 18:43:13.597660 (+ 1162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:13.597668 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:13.597890 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:13.598046 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:43:13.599094 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:13.600736 (+ 1642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:13.603143 (+ 2407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:13.604502 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:13.604638 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:13.604851 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:13.605071 (+ 220us) server_negotiation.cc:299] Negotiation successful
0620 18:43:13.605233 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10158,"thread_start_us":10374,"threads_started":1}
I20260620 18:43:13.607731 11138 tablet_service.cc:1467] Processing CreateTablet for tablet 39687191248c4652bcd8bcf1f0ffc260 (DEFAULT_TABLE table=test-table [id=b37edcdc79cc453ebdc8a7ce28105e65]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:43:13.609025 11138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39687191248c4652bcd8bcf1f0ffc260. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:43:13.608745 11004 tablet_service.cc:1467] Processing CreateTablet for tablet 39687191248c4652bcd8bcf1f0ffc260 (DEFAULT_TABLE table=test-table [id=b37edcdc79cc453ebdc8a7ce28105e65]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:43:13.609997 11004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39687191248c4652bcd8bcf1f0ffc260. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:43:13.612269 11352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:13.584124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.3:42473 (local address 127.0.0.1:42414)
0620 18:43:13.601399 (+ 17275us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:13.601416 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:13.601527 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:13.602990 (+ 1463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:13.602994 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:13.603018 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:43:13.603308 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:13.603319 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:13.607722 (+ 4403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:13.607728 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:13.609735 (+ 2007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:13.609751 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:13.611521 (+ 1770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:13.611527 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:13.611693 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:13.611728 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 18:43:13.611820 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 18:43:13.611950 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17131,"thread_start_us":17496,"threads_started":1}
I20260620 18:43:13.612912 11353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:13.584381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42414 (local address 127.23.115.3:42473)
0620 18:43:13.602276 (+ 17895us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:13.602284 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:13.602315 (+ 31us) server_negotiation.cc:407] Connection header received
0620 18:43:13.602402 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:13.602409 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:13.602603 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:13.602799 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:43:13.603633 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:13.607532 (+ 3899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:13.609915 (+ 2383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:13.611321 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:13.611447 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:13.611653 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:13.611890 (+ 237us) server_negotiation.cc:299] Negotiation successful
0620 18:43:13.612046 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17734,"thread_start_us":17588,"threads_started":1}
I20260620 18:43:13.616492 11271 tablet_service.cc:1467] Processing CreateTablet for tablet 39687191248c4652bcd8bcf1f0ffc260 (DEFAULT_TABLE table=test-table [id=b37edcdc79cc453ebdc8a7ce28105e65]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:43:13.617892 11271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39687191248c4652bcd8bcf1f0ffc260. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:43:13.621096 11354 tablet_bootstrap.cc:492] T 39687191248c4652bcd8bcf1f0ffc260 P 4b21af82df0c496eb1af09e4a6dac4cf: Bootstrap starting.
I20260620 18:43:13.624262 11354 tablet_bootstrap.cc:654] T 39687191248c4652bcd8bcf1f0ffc260 P 4b21af82df0c496eb1af09e4a6dac4cf: Neither blocks nor log segments found. Creating new log.
I20260620 18:43:13.625077 11355 tablet_bootstrap.cc:492] T 39687191248c4652bcd8bcf1f0ffc260 P 319393d00cc24530b55827f1521cdc36: Bootstrap starting.
I20260620 18:43:13.625847 11354 log.cc:824] T 39687191248c4652bcd8bcf1f0ffc260 P 4b21af82df0c496eb1af09e4a6dac4cf: Log is configured to *not* fsync() on all Append() calls
I20260620 18:43:13.628432 11355 tablet_bootstrap.cc:654] T 39687191248c4652bcd8bcf1f0ffc260 P 319393d00cc24530b55827f1521cdc36: Neither blocks nor log segments found. Creating new log.
I20260620 18:43:13.629249 11354 tablet_bootstrap.cc:492] T 39687191248c4652bcd8bcf1f0ffc260 P 4b21af82df0c496eb1af09e4a6dac4cf: No bootstrap required, opened a new log
I20260620 18:43:13.629523 11354 ts_tablet_manager.cc:1397] T 39687191248c4652bcd8bcf1f0ffc260 P 4b21af82df0c496eb1af09e4a6dac4cf: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260620 18:43:13.629592 11355 log.cc:824] T 39687191248c4652bcd8bcf1f0ffc260 P 319393d00cc24530b55827f1521cdc36: Log is configured to *not* fsync() on all Append() calls
I20260620 18:43:13.632447 11355 tablet_bootstrap.cc:492] T 39687191248c4652bcd8bcf1f0ffc260 P 319393d00cc24530b55827f1521cdc36: No bootstrap required, opened a new log
I20260620 18:43:13.632689 11355 ts_tablet_manager.cc:1397] T 39687191248c4652bcd8bcf1f0ffc260 P 319393d00cc24530b55827f1521cdc36: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260620 18:43:13.632738 11358 tablet_bootstrap.cc:492] T 39687191248c4652bcd8bcf1f0ffc260 P e32dd350c9af47c2a8a461e504f9188a: Bootstrap starting.
I20260620 18:43:13.635607 11358 tablet_bootstrap.cc:654] T 39687191248c4652bcd8bcf1f0ffc260 P e32dd350c9af47c2a8a461e504f9188a: Neither blocks nor log segments found. Creating new log.
I20260620 18:43:13.636509 11358 log.cc:824] T 39687191248c4652bcd8bcf1f0ffc260 P e32dd350c9af47c2a8a461e504f9188a: Log is configured to *not* fsync() on all Append() calls
I20260620 18:43:13.638768 11358 tablet_bootstrap.cc:492] T 39687191248c4652bcd8bcf1f0ffc260 P e32dd350c9af47c2a8a461e504f9188a: No bootstrap required, opened a new log
I20260620 18:43:13.639024 11358 ts_tablet_manager.cc:1397] T 39687191248c4652bcd8bcf1f0ffc260 P e32dd350c9af47c2a8a461e504f9188a: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260620 18:43:13.639509 11354 raft_consensus.cc:348] T 39687191248c4652bcd8bcf1f0ffc260 P 4b21af82df0c496eb1af09e4a6dac4cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "319393d00cc24530b55827f1521cdc36" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 33641 } } peers { permanent_uuid: "4b21af82df0c496eb1af09e4a6dac4cf" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 46291 } } peers { permanent_uuid: "e32dd350c9af47c2a8a461e504f9188a" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 42473 } }
I20260620 18:43:13.639926 11354 raft_consensus.cc:374] T 39687191248c4652bcd8bcf1f0ffc260 P 4b21af82df0c496eb1af09e4a6dac4cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:43:13.640020 11354 raft_consensus.cc:729] T 39687191248c4652bcd8bcf1f0ffc260 P 4b21af82df0c496eb1af09e4a6dac4cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b21af82df0c496eb1af09e4a6dac4cf, State: Initialized, Role: FOLLOWER
I20260620 18:43:13.640638 11354 consensus_queue.cc:260] T 39687191248c4652bcd8bcf1f0ffc260 P 4b21af82df0c496eb1af09e4a6dac4cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "319393d00cc24530b55827f1521cdc36" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 33641 } } peers { permanent_uuid: "4b21af82df0c496eb1af09e4a6dac4cf" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 46291 } } peers { permanent_uuid: "e32dd350c9af47c2a8a461e504f9188a" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 42473 } }
I20260620 18:43:13.642350 11354 ts_tablet_manager.cc:1428] T 39687191248c4652bcd8bcf1f0ffc260 P 4b21af82df0c496eb1af09e4a6dac4cf: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260620 18:43:13.642562 11355 raft_consensus.cc:348] T 39687191248c4652bcd8bcf1f0ffc260 P 319393d00cc24530b55827f1521cdc36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "319393d00cc24530b55827f1521cdc36" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 33641 } } peers { permanent_uuid: "4b21af82df0c496eb1af09e4a6dac4cf" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 46291 } } peers { permanent_uuid: "e32dd350c9af47c2a8a461e504f9188a" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 42473 } }
I20260620 18:43:13.643005 11355 raft_consensus.cc:374] T 39687191248c4652bcd8bcf1f0ffc260 P 319393d00cc24530b55827f1521cdc36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:43:13.643080 11355 raft_consensus.cc:729] T 39687191248c4652bcd8bcf1f0ffc260 P 319393d00cc24530b55827f1521cdc36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 319393d00cc24530b55827f1521cdc36, State: Initialized, Role: FOLLOWER
I20260620 18:43:13.643592 11355 consensus_queue.cc:260] T 39687191248c4652bcd8bcf1f0ffc260 P 319393d00cc24530b55827f1521cdc36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "319393d00cc24530b55827f1521cdc36" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 33641 } } peers { permanent_uuid: "4b21af82df0c496eb1af09e4a6dac4cf" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 46291 } } peers { permanent_uuid: "e32dd350c9af47c2a8a461e504f9188a" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 42473 } }
I20260620 18:43:13.645175 11355 ts_tablet_manager.cc:1428] T 39687191248c4652bcd8bcf1f0ffc260 P 319393d00cc24530b55827f1521cdc36: Time spent starting tablet: real 0.012s user 0.009s sys 0.002s
I20260620 18:43:13.645562 11203 heartbeater.cc:503] Master 127.23.115.62:43451 was elected leader, sending a full tablet report...
I20260620 18:43:13.647565 11358 raft_consensus.cc:348] T 39687191248c4652bcd8bcf1f0ffc260 P e32dd350c9af47c2a8a461e504f9188a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "319393d00cc24530b55827f1521cdc36" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 33641 } } peers { permanent_uuid: "4b21af82df0c496eb1af09e4a6dac4cf" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 46291 } } peers { permanent_uuid: "e32dd350c9af47c2a8a461e504f9188a" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 42473 } }
I20260620 18:43:13.647969 11358 raft_consensus.cc:374] T 39687191248c4652bcd8bcf1f0ffc260 P e32dd350c9af47c2a8a461e504f9188a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:43:13.648067 11358 raft_consensus.cc:729] T 39687191248c4652bcd8bcf1f0ffc260 P e32dd350c9af47c2a8a461e504f9188a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e32dd350c9af47c2a8a461e504f9188a, State: Initialized, Role: FOLLOWER
I20260620 18:43:13.648517 11358 consensus_queue.cc:260] T 39687191248c4652bcd8bcf1f0ffc260 P e32dd350c9af47c2a8a461e504f9188a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "319393d00cc24530b55827f1521cdc36" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 33641 } } peers { permanent_uuid: "4b21af82df0c496eb1af09e4a6dac4cf" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 46291 } } peers { permanent_uuid: "e32dd350c9af47c2a8a461e504f9188a" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 42473 } }
I20260620 18:43:13.649371 11336 heartbeater.cc:503] Master 127.23.115.62:43451 was elected leader, sending a full tablet report...
I20260620 18:43:13.649899 11358 ts_tablet_manager.cc:1428] T 39687191248c4652bcd8bcf1f0ffc260 P e32dd350c9af47c2a8a461e504f9188a: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260620 18:43:13.701139 11362 raft_consensus.cc:482] T 39687191248c4652bcd8bcf1f0ffc260 P e32dd350c9af47c2a8a461e504f9188a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:43:13.701431 11362 raft_consensus.cc:504] T 39687191248c4652bcd8bcf1f0ffc260 P e32dd350c9af47c2a8a461e504f9188a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "319393d00cc24530b55827f1521cdc36" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 33641 } } peers { permanent_uuid: "4b21af82df0c496eb1af09e4a6dac4cf" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 46291 } } peers { permanent_uuid: "e32dd350c9af47c2a8a461e504f9188a" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 42473 } }
I20260620 18:43:13.703117 11362 leader_election.cc:288] T 39687191248c4652bcd8bcf1f0ffc260 P e32dd350c9af47c2a8a461e504f9188a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 319393d00cc24530b55827f1521cdc36 (127.23.115.2:33641), 4b21af82df0c496eb1af09e4a6dac4cf (127.23.115.1:46291)
I20260620 18:43:13.708650 11338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:13.703547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.2:33641 (local address 127.23.115.3:46919)
0620 18:43:13.703908 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:13.703923 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:13.704056 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:13.704411 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:13.704416 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:13.704442 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:43:13.704600 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:13.704607 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:13.705558 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:13.705563 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:13.706833 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:13.706844 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:13.707924 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:13.707930 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:13.708128 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:13.708152 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 18:43:13.708232 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 18:43:13.708355 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224}
I20260620 18:43:13.708650 11350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:13.703581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.3:46919 (local address 127.23.115.2:33641)
0620 18:43:13.703774 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:13.703779 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:13.704000 (+ 221us) server_negotiation.cc:407] Connection header received
0620 18:43:13.704191 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:13.704195 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:13.704270 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:13.704362 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:43:13.704728 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:13.705421 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:13.707002 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:13.707774 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:13.707924 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:13.707998 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:13.708267 (+ 269us) server_negotiation.cc:299] Negotiation successful
0620 18:43:13.708426 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 18:43:13.709321 11351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:13.703558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.3:51581 (local address 127.23.115.1:46291)
0620 18:43:13.703767 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:13.703774 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:13.704366 (+ 592us) server_negotiation.cc:407] Connection header received
0620 18:43:13.704550 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:13.704554 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:13.704611 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:13.704684 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:43:13.705176 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:13.705893 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:13.707608 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:13.708380 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:13.708519 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:13.708586 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:13.708956 (+ 370us) server_negotiation.cc:299] Negotiation successful
0620 18:43:13.709090 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 18:43:13.709391 11363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:13.703575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.1:46291 (local address 127.23.115.3:51581)
0620 18:43:13.704273 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:13.704291 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:13.704423 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:13.704710 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:13.704715 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:13.704743 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:43:13.705033 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:13.705041 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:13.706037 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:13.706043 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:13.707457 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:13.707469 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:13.708566 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:13.708572 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:13.708717 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:13.708748 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 18:43:13.708893 (+ 145us) client_negotiation.cc:240] Negotiation successful
0620 18:43:13.709025 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"mutex_wait_us":5,"thread_start_us":408,"threads_started":1}
I20260620 18:43:13.709805 11158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39687191248c4652bcd8bcf1f0ffc260" candidate_uuid: "e32dd350c9af47c2a8a461e504f9188a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "319393d00cc24530b55827f1521cdc36" is_pre_election: true
I20260620 18:43:13.710289 11158 raft_consensus.cc:2454] T 39687191248c4652bcd8bcf1f0ffc260 P 319393d00cc24530b55827f1521cdc36 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e32dd350c9af47c2a8a461e504f9188a in term 0.
I20260620 18:43:13.710273 11024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39687191248c4652bcd8bcf1f0ffc260" candidate_uuid: "e32dd350c9af47c2a8a461e504f9188a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b21af82df0c496eb1af09e4a6dac4cf" is_pre_election: true
I20260620 18:43:13.710791 11024 raft_consensus.cc:2454] T 39687191248c4652bcd8bcf1f0ffc260 P 4b21af82df0c496eb1af09e4a6dac4cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e32dd350c9af47c2a8a461e504f9188a in term 0.
I20260620 18:43:13.710915 11224 leader_election.cc:302] T 39687191248c4652bcd8bcf1f0ffc260 P e32dd350c9af47c2a8a461e504f9188a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 319393d00cc24530b55827f1521cdc36, e32dd350c9af47c2a8a461e504f9188a; no voters:
I20260620 18:43:13.711282 11362 raft_consensus.cc:2781] T 39687191248c4652bcd8bcf1f0ffc260 P e32dd350c9af47c2a8a461e504f9188a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:43:13.711411 11362 raft_consensus.cc:482] T 39687191248c4652bcd8bcf1f0ffc260 P e32dd350c9af47c2a8a461e504f9188a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:43:13.711506 11362 raft_consensus.cc:3037] T 39687191248c4652bcd8bcf1f0ffc260 P e32dd350c9af47c2a8a461e504f9188a [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:43:13.713589 11362 raft_consensus.cc:504] T 39687191248c4652bcd8bcf1f0ffc260 P e32dd350c9af47c2a8a461e504f9188a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "319393d00cc24530b55827f1521cdc36" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 33641 } } peers { permanent_uuid: "4b21af82df0c496eb1af09e4a6dac4cf" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 46291 } } peers { permanent_uuid: "e32dd350c9af47c2a8a461e504f9188a" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 42473 } }
I20260620 18:43:13.714174 11362 leader_election.cc:288] T 39687191248c4652bcd8bcf1f0ffc260 P e32dd350c9af47c2a8a461e504f9188a [CANDIDATE]: Term 1 election: Requested vote from peers 319393d00cc24530b55827f1521cdc36 (127.23.115.2:33641), 4b21af82df0c496eb1af09e4a6dac4cf (127.23.115.1:46291)
I20260620 18:43:13.714627 11158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39687191248c4652bcd8bcf1f0ffc260" candidate_uuid: "e32dd350c9af47c2a8a461e504f9188a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "319393d00cc24530b55827f1521cdc36"
I20260620 18:43:13.714663 11024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39687191248c4652bcd8bcf1f0ffc260" candidate_uuid: "e32dd350c9af47c2a8a461e504f9188a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b21af82df0c496eb1af09e4a6dac4cf"
I20260620 18:43:13.714805 11158 raft_consensus.cc:3037] T 39687191248c4652bcd8bcf1f0ffc260 P 319393d00cc24530b55827f1521cdc36 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:43:13.714855 11024 raft_consensus.cc:3037] T 39687191248c4652bcd8bcf1f0ffc260 P 4b21af82df0c496eb1af09e4a6dac4cf [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:43:13.716923 11158 raft_consensus.cc:2454] T 39687191248c4652bcd8bcf1f0ffc260 P 319393d00cc24530b55827f1521cdc36 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e32dd350c9af47c2a8a461e504f9188a in term 1.
I20260620 18:43:13.717283 11224 leader_election.cc:302] T 39687191248c4652bcd8bcf1f0ffc260 P e32dd350c9af47c2a8a461e504f9188a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 319393d00cc24530b55827f1521cdc36, e32dd350c9af47c2a8a461e504f9188a; no voters:
I20260620 18:43:13.717293 11024 raft_consensus.cc:2454] T 39687191248c4652bcd8bcf1f0ffc260 P 4b21af82df0c496eb1af09e4a6dac4cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e32dd350c9af47c2a8a461e504f9188a in term 1.
I20260620 18:43:13.717525 11362 raft_consensus.cc:2781] T 39687191248c4652bcd8bcf1f0ffc260 P e32dd350c9af47c2a8a461e504f9188a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:43:13.718115 11362 raft_consensus.cc:686] T 39687191248c4652bcd8bcf1f0ffc260 P e32dd350c9af47c2a8a461e504f9188a [term 1 LEADER]: Becoming Leader. State: Replica: e32dd350c9af47c2a8a461e504f9188a, State: Running, Role: LEADER
I20260620 18:43:13.718637 11362 consensus_queue.cc:237] T 39687191248c4652bcd8bcf1f0ffc260 P e32dd350c9af47c2a8a461e504f9188a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "319393d00cc24530b55827f1521cdc36" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 33641 } } peers { permanent_uuid: "4b21af82df0c496eb1af09e4a6dac4cf" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 46291 } } peers { permanent_uuid: "e32dd350c9af47c2a8a461e504f9188a" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 42473 } }
I20260620 18:43:13.723397 10879 catalog_manager.cc:5495] T 39687191248c4652bcd8bcf1f0ffc260 P e32dd350c9af47c2a8a461e504f9188a reported cstate change: term changed from 0 to 1, leader changed from <none> to e32dd350c9af47c2a8a461e504f9188a (127.23.115.3). New cstate: current_term: 1 leader_uuid: "e32dd350c9af47c2a8a461e504f9188a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "319393d00cc24530b55827f1521cdc36" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 33641 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b21af82df0c496eb1af09e4a6dac4cf" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 46291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e32dd350c9af47c2a8a461e504f9188a" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 42473 } health_report { overall_health: HEALTHY } } }
W20260620 18:43:13.758409 11337 tablet.cc:2369] T 39687191248c4652bcd8bcf1f0ffc260 P e32dd350c9af47c2a8a461e504f9188a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:43:13.760638 11353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:13.755476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42422 (local address 127.23.115.3:42473)
0620 18:43:13.755643 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:13.755647 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:13.755664 (+ 17us) server_negotiation.cc:407] Connection header received
0620 18:43:13.755828 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:13.755832 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:13.755931 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:13.756048 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:43:13.756456 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:13.757213 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:13.758679 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:13.759122 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:13.759215 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:13.759689 (+ 474us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:43:13.760194 (+ 505us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:43:13.760307 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:13.760389 (+ 82us) server_negotiation.cc:299] Negotiation successful
0620 18:43:13.760455 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 18:43:13.768573 11024 raft_consensus.cc:1261] T 39687191248c4652bcd8bcf1f0ffc260 P 4b21af82df0c496eb1af09e4a6dac4cf [term 1 FOLLOWER]: Refusing update from remote peer e32dd350c9af47c2a8a461e504f9188a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:43:13.768952 11158 raft_consensus.cc:1261] T 39687191248c4652bcd8bcf1f0ffc260 P 319393d00cc24530b55827f1521cdc36 [term 1 FOLLOWER]: Refusing update from remote peer e32dd350c9af47c2a8a461e504f9188a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:43:13.769553 11364 consensus_queue.cc:1035] T 39687191248c4652bcd8bcf1f0ffc260 P e32dd350c9af47c2a8a461e504f9188a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b21af82df0c496eb1af09e4a6dac4cf" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 46291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:43:13.769968 11362 consensus_queue.cc:1035] T 39687191248c4652bcd8bcf1f0ffc260 P e32dd350c9af47c2a8a461e504f9188a [LEADER]: Connected to new peer: Peer: permanent_uuid: "319393d00cc24530b55827f1521cdc36" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 33641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 18:43:13.773744 11204 tablet.cc:2369] T 39687191248c4652bcd8bcf1f0ffc260 P 319393d00cc24530b55827f1521cdc36: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:43:13.781746 11070 tablet.cc:2369] T 39687191248c4652bcd8bcf1f0ffc260 P 4b21af82df0c496eb1af09e4a6dac4cf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:43:13.793992 10880 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48680:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:43:13.794286 10880 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48680:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:43:13.798697 10880 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 285063a052d2448da228e29fd3e3bd1e: Sending DeleteTablet for 3 replicas of tablet 39687191248c4652bcd8bcf1f0ffc260
I20260620 18:43:13.799727 11004 tablet_service.cc:1514] Processing DeleteTablet for tablet 39687191248c4652bcd8bcf1f0ffc260 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:43:13 UTC) from {username='kudu', principal='kudu/127.23.115.62@KRBTEST.COM'} at 127.0.0.1:34536
I20260620 18:43:13.799765 11271 tablet_service.cc:1514] Processing DeleteTablet for tablet 39687191248c4652bcd8bcf1f0ffc260 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:43:13 UTC) from {username='kudu', principal='kudu/127.23.115.62@KRBTEST.COM'} at 127.0.0.1:42414
I20260620 18:43:13.799835 11003 tablet_service.cc:1514] Processing DeleteTablet for tablet 39687191248c4652bcd8bcf1f0ffc260 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:43:13 UTC) from {username='kudu', principal='kudu/127.23.115.62@KRBTEST.COM'} at 127.0.0.1:34536
I20260620 18:43:13.799875 11138 tablet_service.cc:1514] Processing DeleteTablet for tablet 39687191248c4652bcd8bcf1f0ffc260 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:43:13 UTC) from {username='kudu', principal='kudu/127.23.115.62@KRBTEST.COM'} at 127.0.0.1:39772
I20260620 18:43:13.800552 11376 tablet_replica.cc:330] T 39687191248c4652bcd8bcf1f0ffc260 P 4b21af82df0c496eb1af09e4a6dac4cf: stopping tablet replica
I20260620 18:43:13.800936 11376 raft_consensus.cc:2229] T 39687191248c4652bcd8bcf1f0ffc260 P 4b21af82df0c496eb1af09e4a6dac4cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:43:13.801373 11376 raft_consensus.cc:2258] T 39687191248c4652bcd8bcf1f0ffc260 P 4b21af82df0c496eb1af09e4a6dac4cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:43:13.801608 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 10941
I20260620 18:43:13.802242 11377 tablet_replica.cc:330] T 39687191248c4652bcd8bcf1f0ffc260 P e32dd350c9af47c2a8a461e504f9188a: stopping tablet replica
I20260620 18:43:13.802606 11377 raft_consensus.cc:2229] T 39687191248c4652bcd8bcf1f0ffc260 P e32dd350c9af47c2a8a461e504f9188a [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:43:13.803258 11377 raft_consensus.cc:2258] T 39687191248c4652bcd8bcf1f0ffc260 P e32dd350c9af47c2a8a461e504f9188a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:43:13.805212 11376 ts_tablet_manager.cc:1905] T 39687191248c4652bcd8bcf1f0ffc260 P 4b21af82df0c496eb1af09e4a6dac4cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:43:13.812458 11376 ts_tablet_manager.cc:1918] T 39687191248c4652bcd8bcf1f0ffc260 P 4b21af82df0c496eb1af09e4a6dac4cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:43:13.812613 11376 log.cc:1196] T 39687191248c4652bcd8bcf1f0ffc260 P 4b21af82df0c496eb1af09e4a6dac4cf: Deleting WAL directory at /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980676253068-24012-0/minicluster-data/ts-0/wal/wals/39687191248c4652bcd8bcf1f0ffc260
I20260620 18:43:13.813150 11376 ts_tablet_manager.cc:1939] T 39687191248c4652bcd8bcf1f0ffc260 P 4b21af82df0c496eb1af09e4a6dac4cf: Deleting consensus metadata
I20260620 18:43:13.815376 10867 catalog_manager.cc:4841] TS 4b21af82df0c496eb1af09e4a6dac4cf (127.23.115.1:46291): tablet 39687191248c4652bcd8bcf1f0ffc260 (table test-table [id=b37edcdc79cc453ebdc8a7ce28105e65]) successfully deleted
W20260620 18:43:13.816377 10867 catalog_manager.cc:4808] TS 4b21af82df0c496eb1af09e4a6dac4cf (127.23.115.1:46291): delete failed for tablet 39687191248c4652bcd8bcf1f0ffc260 because the tablet was not found. No further retry: Not found: Tablet not found: 39687191248c4652bcd8bcf1f0ffc260
I20260620 18:43:13.816910 11377 ts_tablet_manager.cc:1905] T 39687191248c4652bcd8bcf1f0ffc260 P e32dd350c9af47c2a8a461e504f9188a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:43:13.818001 11378 tablet_replica.cc:330] T 39687191248c4652bcd8bcf1f0ffc260 P 319393d00cc24530b55827f1521cdc36: stopping tablet replica
I20260620 18:43:13.818439 11378 raft_consensus.cc:2229] T 39687191248c4652bcd8bcf1f0ffc260 P 319393d00cc24530b55827f1521cdc36 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:43:13.818877 11378 raft_consensus.cc:2258] T 39687191248c4652bcd8bcf1f0ffc260 P 319393d00cc24530b55827f1521cdc36 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:43:13.820555 11378 ts_tablet_manager.cc:1905] T 39687191248c4652bcd8bcf1f0ffc260 P 319393d00cc24530b55827f1521cdc36: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 20 18:43:13 dist-test-slave-d645 krb5kdc[10835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980991, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.115.1@KRBTEST.COM
I20260620 18:43:13.823045 11377 ts_tablet_manager.cc:1918] T 39687191248c4652bcd8bcf1f0ffc260 P e32dd350c9af47c2a8a461e504f9188a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:43:13.823220 11377 log.cc:1196] T 39687191248c4652bcd8bcf1f0ffc260 P e32dd350c9af47c2a8a461e504f9188a: Deleting WAL directory at /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980676253068-24012-0/minicluster-data/ts-2/wal/wals/39687191248c4652bcd8bcf1f0ffc260
I20260620 18:43:13.823776 11377 ts_tablet_manager.cc:1939] T 39687191248c4652bcd8bcf1f0ffc260 P e32dd350c9af47c2a8a461e504f9188a: Deleting consensus metadata
I20260620 18:43:13.825577 10865 catalog_manager.cc:4841] TS e32dd350c9af47c2a8a461e504f9188a (127.23.115.3:42473): tablet 39687191248c4652bcd8bcf1f0ffc260 (table test-table [id=b37edcdc79cc453ebdc8a7ce28105e65]) successfully deleted
I20260620 18:43:13.827591 11378 ts_tablet_manager.cc:1918] T 39687191248c4652bcd8bcf1f0ffc260 P 319393d00cc24530b55827f1521cdc36: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:43:13.827759 11378 log.cc:1196] T 39687191248c4652bcd8bcf1f0ffc260 P 319393d00cc24530b55827f1521cdc36: Deleting WAL directory at /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980676253068-24012-0/minicluster-data/ts-1/wal/wals/39687191248c4652bcd8bcf1f0ffc260
I20260620 18:43:13.828258 11378 ts_tablet_manager.cc:1939] T 39687191248c4652bcd8bcf1f0ffc260 P 319393d00cc24530b55827f1521cdc36: Deleting consensus metadata
I20260620 18:43:13.829852 10864 catalog_manager.cc:4841] TS 319393d00cc24530b55827f1521cdc36 (127.23.115.2:33641): tablet 39687191248c4652bcd8bcf1f0ffc260 (table test-table [id=b37edcdc79cc453ebdc8a7ce28105e65]) successfully deleted
I20260620 18:43:13.830708 11351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:13.802977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34538 (local address 127.23.115.1:46291)
0620 18:43:13.803141 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:13.803148 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:13.812757 (+ 9609us) server_negotiation.cc:407] Connection header received
0620 18:43:13.813223 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:13.813229 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:13.813332 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:13.813405 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:13.815100 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:13.816129 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:13.817553 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:13.818130 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:13.818245 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:13.823435 (+ 5190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:13.823452 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:13.823468 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:13.823515 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:13.827386 (+ 3871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:13.828338 (+ 952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:13.828347 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:13.828351 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:13.828516 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:13.828962 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:13.828967 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:13.828971 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:13.829452 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:13.829776 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:13.830098 (+ 322us) server_negotiation.cc:299] Negotiation successful
0620 18:43:13.830329 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 18:43:13.831476 11064 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:43:13.946532 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 10941
I20260620 18:43:13.961683 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 11075
Jun 20 18:43:13 dist-test-slave-d645 krb5kdc[10835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980991, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.115.2@KRBTEST.COM
I20260620 18:43:13.978669 11350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:13.962414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39776 (local address 127.23.115.2:33641)
0620 18:43:13.962608 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:13.962614 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:13.962635 (+ 21us) server_negotiation.cc:407] Connection header received
0620 18:43:13.962716 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:13.962722 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:13.962804 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:13.962915 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:13.964233 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:13.965447 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:13.966981 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:13.967583 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:13.967722 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:13.972068 (+ 4346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:13.972086 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:13.972107 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:13.972155 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:13.975864 (+ 3709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:13.976604 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:13.976615 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:13.976618 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:13.976751 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:13.977101 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:13.977106 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:13.977110 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:13.977526 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:13.977810 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:13.978139 (+ 329us) server_negotiation.cc:299] Negotiation successful
0620 18:43:13.978364 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 18:43:13.979413 11198 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:43:14.079093 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 11075
I20260620 18:43:14.094311 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 11208
Jun 20 18:43:14 dist-test-slave-d645 krb5kdc[10835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980991, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.115.3@KRBTEST.COM
I20260620 18:43:14.109824 11353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:14.095017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42426 (local address 127.23.115.3:42473)
0620 18:43:14.095218 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:14.095223 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:14.095370 (+ 147us) server_negotiation.cc:407] Connection header received
0620 18:43:14.095542 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:14.095547 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:14.095616 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:14.095708 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:14.096967 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:14.097747 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:14.099014 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:14.099478 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:14.099590 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:14.103897 (+ 4307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:14.103909 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:14.103927 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:14.103969 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:14.107104 (+ 3135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:14.107706 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:14.107713 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:14.107719 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:14.107860 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:14.108199 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:14.108203 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:14.108205 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:14.108555 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:14.108847 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:14.109135 (+ 288us) server_negotiation.cc:299] Negotiation successful
0620 18:43:14.109383 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 18:43:14.110459 11331 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:43:14.231941 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 11208
I20260620 18:43:14.247617 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 10848
I20260620 18:43:14.248636 10910 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:43:14.357240 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 10848
2026-06-20T18:43:14Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4136 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-20T18:43:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:43:14Z Disabled control of system clock
I20260620 18:43:14.427215 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980676253068-24012-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980676253068-24012-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980676253068-24012-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.115.62:40877
--webserver_interface=127.23.115.62
--webserver_port=0
--builtin_ntp_servers=127.23.115.20:39579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.115.62:40877
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 18:43:14.741644 11387 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:43:14.741926 11387 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:43:14.742003 11387 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:43:14.749154 11387 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:43:14.749267 11387 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:43:14.749305 11387 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:43:14.749361 11387 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:43:14.749408 11387 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:43:14.758637 11387 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:39579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980676253068-24012-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980676253068-24012-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.115.62:40877
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.115.62:40877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.115.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980676253068-24012-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:43:14.759917 11387 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:43:14.761569 11387 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:43:14.772928 11395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:14.772990 11392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:14.772964 11393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:14.773802 11387 server_base.cc:1034] running on GCE node
I20260620 18:43:14.774605 11387 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:43:14.776037 11387 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:43:14.777289 11387 hybrid_clock.cc:648] HybridClock initialized: now 1781980994777227 us; error 32 us; skew 500 ppm
I20260620 18:43:14.777611 11387 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:43:14.779862 11387 webserver.cc:460] Webserver started at http://127.23.115.62:41489/ using document root <none> and password file <none>
I20260620 18:43:14.780514 11387 fs_manager.cc:359] Metadata directory not provided
I20260620 18:43:14.780598 11387 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:43:14.780851 11387 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:43:14.783704 11387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980676253068-24012-0/minicluster-data/master-0/data/instance:
uuid: "a3c7620f7e0a4afb9a6371cc242a4b09"
format_stamp: "Formatted at 2026-06-20 18:43:14 on dist-test-slave-d645"
server_key: "6ea7516b62f8e5e248d21d6fbd31ec0b"
server_key_iv: "2fa539a644dbcba4dec30e586016d96e"
server_key_version: "encryptionkey@0"
I20260620 18:43:14.784439 11387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980676253068-24012-0/minicluster-data/master-0/wal/instance:
uuid: "a3c7620f7e0a4afb9a6371cc242a4b09"
format_stamp: "Formatted at 2026-06-20 18:43:14 on dist-test-slave-d645"
server_key: "6ea7516b62f8e5e248d21d6fbd31ec0b"
server_key_iv: "2fa539a644dbcba4dec30e586016d96e"
server_key_version: "encryptionkey@0"
I20260620 18:43:14.789229 11387 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 18:43:14.792598 11401 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:14.793864 11387 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 18:43:14.794009 11387 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980676253068-24012-0/minicluster-data/master-0/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980676253068-24012-0/minicluster-data/master-0/wal
uuid: "a3c7620f7e0a4afb9a6371cc242a4b09"
format_stamp: "Formatted at 2026-06-20 18:43:14 on dist-test-slave-d645"
server_key: "6ea7516b62f8e5e248d21d6fbd31ec0b"
server_key_iv: "2fa539a644dbcba4dec30e586016d96e"
server_key_version: "encryptionkey@0"
I20260620 18:43:14.794189 11387 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980676253068-24012-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980676253068-24012-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980676253068-24012-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:43:14.819231 11387 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:43:14.820106 11387 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:43:14.820350 11387 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:43:14.837608 11387 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.62:40877
I20260620 18:43:14.837617 11452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.62:40877 every 8 connection(s)
I20260620 18:43:14.839141 11387 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
I20260620 18:43:14.842783 11453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:43:14.848731 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 11387
I20260620 18:43:14.848896 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980676253068-24012-0/minicluster-data/master-0/wal/instance
I20260620 18:43:14.849339 24012 external_mini_cluster.cc:1428] Setting key 448d7b4148d2cfc862f83745971bc621
I20260620 18:43:14.853123 11453 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3c7620f7e0a4afb9a6371cc242a4b09: Bootstrap starting.
I20260620 18:43:14.856323 11453 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a3c7620f7e0a4afb9a6371cc242a4b09: Neither blocks nor log segments found. Creating new log.
I20260620 18:43:14.857314 11453 log.cc:824] T 00000000000000000000000000000000 P a3c7620f7e0a4afb9a6371cc242a4b09: Log is configured to *not* fsync() on all Append() calls
I20260620 18:43:14.859894 11453 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3c7620f7e0a4afb9a6371cc242a4b09: No bootstrap required, opened a new log
I20260620 18:43:14.861900 11456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:14.851120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44630 (local address 127.23.115.62:40877)
0620 18:43:14.852027 (+ 907us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:14.852037 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:14.852078 (+ 41us) server_negotiation.cc:407] Connection header received
0620 18:43:14.853756 (+ 1678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:14.853777 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:14.854185 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:14.854505 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:14.855173 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:14.856412 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:14.857850 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:14.858523 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:14.858683 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:14.859734 (+ 1051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:14.859741 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:14.859754 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:43:14.860301 (+ 547us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:14.860385 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:43:14.860409 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:14.860507 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:14.860726 (+ 219us) server_negotiation.cc:299] Negotiation successful
0620 18:43:14.860997 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":312,"threads_started":1}
I20260620 18:43:14.867343 11453 raft_consensus.cc:348] T 00000000000000000000000000000000 P a3c7620f7e0a4afb9a6371cc242a4b09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3c7620f7e0a4afb9a6371cc242a4b09" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 40877 } }
I20260620 18:43:14.867667 11453 raft_consensus.cc:374] T 00000000000000000000000000000000 P a3c7620f7e0a4afb9a6371cc242a4b09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:43:14.867751 11453 raft_consensus.cc:729] T 00000000000000000000000000000000 P a3c7620f7e0a4afb9a6371cc242a4b09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3c7620f7e0a4afb9a6371cc242a4b09, State: Initialized, Role: FOLLOWER
I20260620 18:43:14.868305 11453 consensus_queue.cc:260] T 00000000000000000000000000000000 P a3c7620f7e0a4afb9a6371cc242a4b09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3c7620f7e0a4afb9a6371cc242a4b09" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 40877 } }
I20260620 18:43:14.868574 11453 raft_consensus.cc:388] T 00000000000000000000000000000000 P a3c7620f7e0a4afb9a6371cc242a4b09 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:43:14.868710 11453 raft_consensus.cc:482] T 00000000000000000000000000000000 P a3c7620f7e0a4afb9a6371cc242a4b09 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:43:14.868902 11453 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a3c7620f7e0a4afb9a6371cc242a4b09 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:43:14.871049 11453 raft_consensus.cc:504] T 00000000000000000000000000000000 P a3c7620f7e0a4afb9a6371cc242a4b09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3c7620f7e0a4afb9a6371cc242a4b09" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 40877 } }
I20260620 18:43:14.871479 11453 leader_election.cc:302] T 00000000000000000000000000000000 P a3c7620f7e0a4afb9a6371cc242a4b09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a3c7620f7e0a4afb9a6371cc242a4b09; no voters:
I20260620 18:43:14.871994 11453 leader_election.cc:288] T 00000000000000000000000000000000 P a3c7620f7e0a4afb9a6371cc242a4b09 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:43:14.872159 11458 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a3c7620f7e0a4afb9a6371cc242a4b09 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:43:14.873566 11458 raft_consensus.cc:686] T 00000000000000000000000000000000 P a3c7620f7e0a4afb9a6371cc242a4b09 [term 1 LEADER]: Becoming Leader. State: Replica: a3c7620f7e0a4afb9a6371cc242a4b09, State: Running, Role: LEADER
I20260620 18:43:14.874087 11458 consensus_queue.cc:237] T 00000000000000000000000000000000 P a3c7620f7e0a4afb9a6371cc242a4b09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3c7620f7e0a4afb9a6371cc242a4b09" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 40877 } }
I20260620 18:43:14.874377 11453 sys_catalog.cc:564] T 00000000000000000000000000000000 P a3c7620f7e0a4afb9a6371cc242a4b09 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:43:14.876827 11460 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3c7620f7e0a4afb9a6371cc242a4b09 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3c7620f7e0a4afb9a6371cc242a4b09. Latest consensus state: current_term: 1 leader_uuid: "a3c7620f7e0a4afb9a6371cc242a4b09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3c7620f7e0a4afb9a6371cc242a4b09" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 40877 } } }
I20260620 18:43:14.876819 11459 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3c7620f7e0a4afb9a6371cc242a4b09 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a3c7620f7e0a4afb9a6371cc242a4b09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3c7620f7e0a4afb9a6371cc242a4b09" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 40877 } } }
I20260620 18:43:14.877321 11459 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3c7620f7e0a4afb9a6371cc242a4b09 [sys.catalog]: This master's current role is: LEADER
I20260620 18:43:14.877926 11460 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3c7620f7e0a4afb9a6371cc242a4b09 [sys.catalog]: This master's current role is: LEADER
I20260620 18:43:14.880151 11464 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:43:14.885656 11464 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:43:14.894534 11464 catalog_manager.cc:1347] Generated new cluster ID: 419a46f1af744aff8a4aec36c07f4ba9
I20260620 18:43:14.894654 11464 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:43:14.911418 11464 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:43:14.912762 11464 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:43:14.921268 11464 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a3c7620f7e0a4afb9a6371cc242a4b09: Generated new TSK 0
I20260620 18:43:14.922117 11464 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:43:14.933372 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980676253068-24012-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980676253068-24012-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980676253068-24012-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980676253068-24012-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.115.1:0
--local_ip_for_outbound_sockets=127.23.115.1
--webserver_interface=127.23.115.1
--webserver_port=0
--tserver_master_addrs=127.23.115.62:40877
--builtin_ntp_servers=127.23.115.20:39579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 18:43:15.276226 11477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:43:15.276528 11477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:43:15.276639 11477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:43:15.284893 11477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:43:15.285012 11477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:43:15.285151 11477 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.115.1
I20260620 18:43:15.294693 11477 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:39579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980676253068-24012-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980676253068-24012-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.115.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980676253068-24012-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.115.1
--webserver_port=0
--tserver_master_addrs=127.23.115.62:40877
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.115.1
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980676253068-24012-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:43:15.296015 11477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:43:15.297748 11477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:43:15.312304 11482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:15.312314 11485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:15.312260 11483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:15.313138 11477 server_base.cc:1034] running on GCE node
I20260620 18:43:15.313750 11477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:43:15.314738 11477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:43:15.315955 11477 hybrid_clock.cc:648] HybridClock initialized: now 1781980995315898 us; error 46 us; skew 500 ppm
I20260620 18:43:15.316291 11477 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:43:15.318840 11477 webserver.cc:460] Webserver started at http://127.23.115.1:33527/ using document root <none> and password file <none>
I20260620 18:43:15.319558 11477 fs_manager.cc:359] Metadata directory not provided
I20260620 18:43:15.319672 11477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:43:15.319976 11477 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:43:15.322890 11477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980676253068-24012-0/minicluster-data/ts-0/data/instance:
uuid: "d118d82c6e6a4e74a69875cb4c85942c"
format_stamp: "Formatted at 2026-06-20 18:43:15 on dist-test-slave-d645"
server_key: "1e3edb4ef6aca8120099094e2037403b"
server_key_iv: "b45ba96fd19cf9b5e6b13a9907c1b2e6"
server_key_version: "encryptionkey@0"
I20260620 18:43:15.323712 11477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980676253068-24012-0/minicluster-data/ts-0/wal/instance:
uuid: "d118d82c6e6a4e74a69875cb4c85942c"
format_stamp: "Formatted at 2026-06-20 18:43:15 on dist-test-slave-d645"
server_key: "1e3edb4ef6aca8120099094e2037403b"
server_key_iv: "b45ba96fd19cf9b5e6b13a9907c1b2e6"
server_key_version: "encryptionkey@0"
I20260620 18:43:15.328742 11477 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:43:15.332351 11491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:15.334020 11477 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:43:15.334185 11477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980676253068-24012-0/minicluster-data/ts-0/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980676253068-24012-0/minicluster-data/ts-0/wal
uuid: "d118d82c6e6a4e74a69875cb4c85942c"
format_stamp: "Formatted at 2026-06-20 18:43:15 on dist-test-slave-d645"
server_key: "1e3edb4ef6aca8120099094e2037403b"
server_key_iv: "b45ba96fd19cf9b5e6b13a9907c1b2e6"
server_key_version: "encryptionkey@0"
I20260620 18:43:15.334405 11477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980676253068-24012-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980676253068-24012-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980676253068-24012-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:43:15.346161 11477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:43:15.347009 11477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:43:15.347290 11477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:43:15.348261 11477 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:43:15.349786 11477 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:43:15.349881 11477 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:15.349990 11477 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:43:15.350062 11477 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:15.382966 11477 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.1:44665
I20260620 18:43:15.383157 11603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.1:44665 every 8 connection(s)
I20260620 18:43:15.384222 11477 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980676253068-24012-0/minicluster-data/ts-0/data/info.pb
I20260620 18:43:15.391213 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 11477
I20260620 18:43:15.391381 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980676253068-24012-0/minicluster-data/ts-0/wal/instance
I20260620 18:43:15.391760 24012 external_mini_cluster.cc:1428] Setting key 3414f164dc8682382ab323640a1d6a11
I20260620 18:43:15.394258 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980676253068-24012-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980676253068-24012-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980676253068-24012-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980676253068-24012-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.115.2:0
--local_ip_for_outbound_sockets=127.23.115.2
--webserver_interface=127.23.115.2
--webserver_port=0
--tserver_master_addrs=127.23.115.62:40877
--builtin_ntp_servers=127.23.115.20:39579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 18:43:15.396756 11606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:15.387186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.1:51099 (local address 127.23.115.62:40877)
0620 18:43:15.387632 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:15.387636 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:15.388741 (+ 1105us) server_negotiation.cc:407] Connection header received
0620 18:43:15.390519 (+ 1778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:15.390524 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:15.390603 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:15.390699 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:15.391802 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:15.392563 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:15.394048 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:15.394495 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:15.394600 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:15.395921 (+ 1321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:15.395928 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:15.395932 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:43:15.395951 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:15.395993 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:43:15.396005 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:15.396079 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:15.396315 (+ 236us) server_negotiation.cc:299] Negotiation successful
0620 18:43:15.396413 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":261,"threads_started":1}
I20260620 18:43:15.398144 11607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:15.387687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:40877 (local address 127.23.115.1:51099)
0620 18:43:15.388578 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:15.388613 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:15.390260 (+ 1647us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:15.390950 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:15.390962 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:15.391061 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:43:15.391642 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:15.391660 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:15.392709 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:15.392713 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:15.393913 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:15.393925 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:15.394713 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:15.394721 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:15.394984 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:15.395617 (+ 633us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:43:15.395620 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:43:15.395662 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:43:15.395740 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:43:15.396091 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:43:15.396103 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:43:15.396111 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 18:43:15.396279 (+ 168us) client_negotiation.cc:240] Negotiation successful
0620 18:43:15.396698 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":280,"threads_started":1}
I20260620 18:43:15.399891 11604 heartbeater.cc:347] Connected to a master server at 127.23.115.62:40877
I20260620 18:43:15.400220 11604 heartbeater.cc:464] Registering TS with master...
I20260620 18:43:15.400941 11604 heartbeater.cc:511] Master 127.23.115.62:40877 requested a full tablet report, sending...
I20260620 18:43:15.402971 11418 ts_manager.cc:194] Registered new tserver with Master: d118d82c6e6a4e74a69875cb4c85942c (127.23.115.1:44665)
I20260620 18:43:15.404451 11418 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.115.1:51099
W20260620 18:43:15.709435 11608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:43:15.709704 11608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:43:15.709839 11608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:43:15.717484 11608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:43:15.717592 11608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:43:15.717757 11608 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.115.2
I20260620 18:43:15.727052 11608 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:39579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980676253068-24012-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980676253068-24012-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.115.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980676253068-24012-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.115.2
--webserver_port=0
--tserver_master_addrs=127.23.115.62:40877
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.115.2
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980676253068-24012-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:43:15.728322 11608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:43:15.730012 11608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:43:15.742914 11616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:15.742892 11614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:15.743319 11608 server_base.cc:1034] running on GCE node
W20260620 18:43:15.743702 11613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:15.744357 11608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:43:15.745316 11608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:43:15.746526 11608 hybrid_clock.cc:648] HybridClock initialized: now 1781980995746472 us; error 41 us; skew 500 ppm
I20260620 18:43:15.746877 11608 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:43:15.749284 11608 webserver.cc:460] Webserver started at http://127.23.115.2:46733/ using document root <none> and password file <none>
I20260620 18:43:15.749992 11608 fs_manager.cc:359] Metadata directory not provided
I20260620 18:43:15.750079 11608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:43:15.750372 11608 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:43:15.753266 11608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980676253068-24012-0/minicluster-data/ts-1/data/instance:
uuid: "ede04a84739f4cad879be7903a016d55"
format_stamp: "Formatted at 2026-06-20 18:43:15 on dist-test-slave-d645"
server_key: "3acbb86d1f0a498e971320a22c566f34"
server_key_iv: "71bbea920d2a4472441fed757ae7e3ba"
server_key_version: "encryptionkey@0"
I20260620 18:43:15.754025 11608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980676253068-24012-0/minicluster-data/ts-1/wal/instance:
uuid: "ede04a84739f4cad879be7903a016d55"
format_stamp: "Formatted at 2026-06-20 18:43:15 on dist-test-slave-d645"
server_key: "3acbb86d1f0a498e971320a22c566f34"
server_key_iv: "71bbea920d2a4472441fed757ae7e3ba"
server_key_version: "encryptionkey@0"
I20260620 18:43:15.759298 11608 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 18:43:15.762888 11622 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:15.764297 11608 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:43:15.764468 11608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980676253068-24012-0/minicluster-data/ts-1/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980676253068-24012-0/minicluster-data/ts-1/wal
uuid: "ede04a84739f4cad879be7903a016d55"
format_stamp: "Formatted at 2026-06-20 18:43:15 on dist-test-slave-d645"
server_key: "3acbb86d1f0a498e971320a22c566f34"
server_key_iv: "71bbea920d2a4472441fed757ae7e3ba"
server_key_version: "encryptionkey@0"
I20260620 18:43:15.764662 11608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980676253068-24012-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980676253068-24012-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980676253068-24012-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:43:15.791658 11608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:43:15.792524 11608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:43:15.792821 11608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:43:15.794023 11608 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:43:15.795974 11608 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:43:15.796068 11608 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:15.796180 11608 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:43:15.796245 11608 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:15.835656 11608 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.2:39531
I20260620 18:43:15.835743 11734 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.2:39531 every 8 connection(s)
I20260620 18:43:15.837023 11608 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980676253068-24012-0/minicluster-data/ts-1/data/info.pb
I20260620 18:43:15.839407 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 11608
I20260620 18:43:15.839592 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980676253068-24012-0/minicluster-data/ts-1/wal/instance
I20260620 18:43:15.840104 24012 external_mini_cluster.cc:1428] Setting key 10e19247352063a4bd390a88067c451e
I20260620 18:43:15.843218 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980676253068-24012-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980676253068-24012-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980676253068-24012-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980676253068-24012-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.115.3:0
--local_ip_for_outbound_sockets=127.23.115.3
--webserver_interface=127.23.115.3
--webserver_port=0
--tserver_master_addrs=127.23.115.62:40877
--builtin_ntp_servers=127.23.115.20:39579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 18:43:15.850380 11606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:15.840414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.2:40723 (local address 127.23.115.62:40877)
0620 18:43:15.840674 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:15.840680 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:15.842396 (+ 1716us) server_negotiation.cc:407] Connection header received
0620 18:43:15.844284 (+ 1888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:15.844289 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:15.844371 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:15.844462 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:15.845480 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:15.846210 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:15.847869 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:15.848266 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:15.848377 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:15.849613 (+ 1236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:15.849619 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:15.849623 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:43:15.849640 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:15.849672 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:43:15.849683 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:15.849754 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:15.849960 (+ 206us) server_negotiation.cc:299] Negotiation successful
0620 18:43:15.850056 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260620 18:43:15.851462 11737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:15.841214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:40877 (local address 127.23.115.2:40723)
0620 18:43:15.842232 (+ 1018us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:15.842275 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:15.844034 (+ 1759us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:15.844679 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:15.844688 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:15.844775 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:43:15.845322 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:15.845339 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:15.846363 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:15.846366 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:15.847733 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:15.847745 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:15.848475 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:15.848482 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:15.848690 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:15.849314 (+ 624us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:43:15.849317 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:43:15.849354 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:43:15.849437 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:43:15.849765 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:43:15.849775 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:43:15.849783 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 18:43:15.849933 (+ 150us) client_negotiation.cc:240] Negotiation successful
0620 18:43:15.850354 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":311,"threads_started":1}
I20260620 18:43:15.852767 11735 heartbeater.cc:347] Connected to a master server at 127.23.115.62:40877
I20260620 18:43:15.853078 11735 heartbeater.cc:464] Registering TS with master...
I20260620 18:43:15.853914 11735 heartbeater.cc:511] Master 127.23.115.62:40877 requested a full tablet report, sending...
I20260620 18:43:15.855520 11418 ts_manager.cc:194] Registered new tserver with Master: ede04a84739f4cad879be7903a016d55 (127.23.115.2:39531)
I20260620 18:43:15.856566 11418 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.115.2:40723
W20260620 18:43:16.171623 11738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:43:16.171880 11738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:43:16.171988 11738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:43:16.179052 11738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:43:16.179147 11738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:43:16.179284 11738 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.115.3
I20260620 18:43:16.188309 11738 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:39579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980676253068-24012-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980676253068-24012-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.115.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980676253068-24012-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.115.3
--webserver_port=0
--tserver_master_addrs=127.23.115.62:40877
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.115.3
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980676253068-24012-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:43:16.189576 11738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:43:16.191239 11738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:43:16.203609 11743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:16.203583 11746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:16.203567 11744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:16.204210 11738 server_base.cc:1034] running on GCE node
I20260620 18:43:16.204756 11738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:43:16.205621 11738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:43:16.206818 11738 hybrid_clock.cc:648] HybridClock initialized: now 1781980996206768 us; error 36 us; skew 500 ppm
I20260620 18:43:16.207139 11738 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:43:16.209414 11738 webserver.cc:460] Webserver started at http://127.23.115.3:37739/ using document root <none> and password file <none>
I20260620 18:43:16.210057 11738 fs_manager.cc:359] Metadata directory not provided
I20260620 18:43:16.210160 11738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:43:16.210456 11738 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:43:16.213254 11738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980676253068-24012-0/minicluster-data/ts-2/data/instance:
uuid: "a4e73df13dd345f78693eca04fba7080"
format_stamp: "Formatted at 2026-06-20 18:43:16 on dist-test-slave-d645"
server_key: "37a0337a57ba5019df73e56ff1aabde0"
server_key_iv: "2c39ea05b760a00f4b6ef9fecf07b8ba"
server_key_version: "encryptionkey@0"
I20260620 18:43:16.214031 11738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980676253068-24012-0/minicluster-data/ts-2/wal/instance:
uuid: "a4e73df13dd345f78693eca04fba7080"
format_stamp: "Formatted at 2026-06-20 18:43:16 on dist-test-slave-d645"
server_key: "37a0337a57ba5019df73e56ff1aabde0"
server_key_iv: "2c39ea05b760a00f4b6ef9fecf07b8ba"
server_key_version: "encryptionkey@0"
I20260620 18:43:16.218878 11738 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 18:43:16.222247 11752 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:16.223805 11738 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:43:16.223958 11738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980676253068-24012-0/minicluster-data/ts-2/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980676253068-24012-0/minicluster-data/ts-2/wal
uuid: "a4e73df13dd345f78693eca04fba7080"
format_stamp: "Formatted at 2026-06-20 18:43:16 on dist-test-slave-d645"
server_key: "37a0337a57ba5019df73e56ff1aabde0"
server_key_iv: "2c39ea05b760a00f4b6ef9fecf07b8ba"
server_key_version: "encryptionkey@0"
I20260620 18:43:16.224162 11738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980676253068-24012-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980676253068-24012-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980676253068-24012-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:43:16.259194 11738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:43:16.260114 11738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:43:16.260438 11738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:43:16.261530 11738 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:43:16.263197 11738 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:43:16.263309 11738 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:16.263412 11738 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:43:16.263478 11738 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:16.296794 11738 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.3:43165
I20260620 18:43:16.296881 11864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.3:43165 every 8 connection(s)
I20260620 18:43:16.298112 11738 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980676253068-24012-0/minicluster-data/ts-2/data/info.pb
I20260620 18:43:16.301815 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 11738
I20260620 18:43:16.301988 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980676253068-24012-0/minicluster-data/ts-2/wal/instance
I20260620 18:43:16.302459 24012 external_mini_cluster.cc:1428] Setting key 1d8a19507d907a33f559cf45db8097ca
I20260620 18:43:16.311621 11606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:16.300831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.3:51085 (local address 127.23.115.62:40877)
0620 18:43:16.301270 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:16.301274 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:16.302869 (+ 1595us) server_negotiation.cc:407] Connection header received
0620 18:43:16.305086 (+ 2217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:16.305094 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:16.305215 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:16.305325 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:16.306449 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:16.307283 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:16.308849 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:16.309237 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:16.309346 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:16.310743 (+ 1397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:16.310749 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:16.310752 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:43:16.310772 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:16.310805 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:43:16.310817 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:16.310909 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:16.311220 (+ 311us) server_negotiation.cc:299] Negotiation successful
0620 18:43:16.311305 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267}
I20260620 18:43:16.312618 11867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:16.301638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:40877 (local address 127.23.115.3:51085)
0620 18:43:16.302697 (+ 1059us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:16.302747 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:16.304584 (+ 1837us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:16.305579 (+ 995us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:16.305588 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:16.305675 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:43:16.306266 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:16.306283 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:16.307458 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:16.307462 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:16.308694 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:16.308706 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:16.309450 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:16.309455 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:16.309663 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:16.310414 (+ 751us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:43:16.310417 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:43:16.310457 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:43:16.310545 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:43:16.310986 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:43:16.310997 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:43:16.311005 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 18:43:16.311162 (+ 157us) client_negotiation.cc:240] Negotiation successful
0620 18:43:16.311489 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":323,"threads_started":1}
I20260620 18:43:16.313918 11865 heartbeater.cc:347] Connected to a master server at 127.23.115.62:40877
I20260620 18:43:16.314193 11865 heartbeater.cc:464] Registering TS with master...
I20260620 18:43:16.314903 11865 heartbeater.cc:511] Master 127.23.115.62:40877 requested a full tablet report, sending...
I20260620 18:43:16.316265 11418 ts_manager.cc:194] Registered new tserver with Master: a4e73df13dd345f78693eca04fba7080 (127.23.115.3:43165)
I20260620 18:43:16.317167 11418 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.115.3:51085
I20260620 18:43:16.317826 24012 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:43:16.327435 11606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:16.321911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44642 (local address 127.23.115.62:40877)
0620 18:43:16.322158 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:16.322164 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:16.322426 (+ 262us) server_negotiation.cc:407] Connection header received
0620 18:43:16.322583 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:16.322589 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:16.322663 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:16.322746 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:16.323268 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:16.323976 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:16.325264 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:16.325662 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:16.325767 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:16.326738 (+ 971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:16.326742 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:16.326746 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:43:16.326762 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:16.326792 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:43:16.326803 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:16.326877 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:16.327051 (+ 174us) server_negotiation.cc:299] Negotiation successful
0620 18:43:16.327141 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 18:43:16.331053 11418 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44642:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:43:16.337993 11418 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:43:16.374605 11877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:16.351075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.2:39531 (local address 127.0.0.1:44280)
0620 18:43:16.365609 (+ 14534us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:16.365627 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:16.365735 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:16.367762 (+ 2027us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:16.367768 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:16.367793 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:43:16.368174 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:16.368187 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:16.369921 (+ 1734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:16.369927 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:16.372094 (+ 2167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:16.372112 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:16.373824 (+ 1712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:16.373831 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:16.373989 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:16.374018 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 18:43:16.374102 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 18:43:16.374220 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14405,"mutex_wait_us":30,"thread_start_us":14758,"threads_started":1}
I20260620 18:43:16.374593 11879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:16.351076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54352 (local address 127.23.115.1:44665)
0620 18:43:16.365234 (+ 14158us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:16.365243 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:16.365277 (+ 34us) server_negotiation.cc:407] Connection header received
0620 18:43:16.365386 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:16.365394 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:16.365656 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:16.365851 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:43:16.367474 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:16.368938 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:16.372350 (+ 3412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:16.373248 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:16.373379 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:16.373624 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:16.374041 (+ 417us) server_negotiation.cc:299] Negotiation successful
0620 18:43:16.374219 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14016,"thread_start_us":13910,"threads_started":1}
I20260620 18:43:16.375304 11878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:16.350865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.1:44665 (local address 127.0.0.1:54352)
0620 18:43:16.364299 (+ 13434us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:16.364356 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:16.364608 (+ 252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:16.366151 (+ 1543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:16.366157 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:16.366181 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:43:16.366602 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:16.366617 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:16.369054 (+ 2437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:16.369060 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:16.372160 (+ 3100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:16.372170 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:16.373428 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:16.373434 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:16.373624 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:16.373677 (+ 53us) client_negotiation.cc:769] Sending connection context
0620 18:43:16.373796 (+ 119us) client_negotiation.cc:240] Negotiation successful
0620 18:43:16.374916 (+ 1120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12981,"mutex_wait_us":2,"thread_start_us":14483,"threads_started":1}
I20260620 18:43:16.376173 11880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:16.351397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44280 (local address 127.23.115.2:39531)
0620 18:43:16.367182 (+ 15785us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:16.367199 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:16.367229 (+ 30us) server_negotiation.cc:407] Connection header received
0620 18:43:16.367317 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:16.367324 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:16.367550 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:16.368418 (+ 868us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:43:16.368675 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:16.369794 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:16.372278 (+ 2484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:16.373638 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:16.373767 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:16.373992 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:16.374189 (+ 197us) server_negotiation.cc:299] Negotiation successful
0620 18:43:16.374491 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15615,"thread_start_us":15479,"threads_started":1}
I20260620 18:43:16.376255 11881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:16.366382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.3:43165 (local address 127.0.0.1:38712)
0620 18:43:16.368029 (+ 1647us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:16.368048 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:16.368334 (+ 286us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:16.369195 (+ 861us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:16.369199 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:16.369223 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:43:16.369647 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:16.369659 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:16.371189 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:16.371195 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:16.373211 (+ 2016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:16.373225 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:16.375513 (+ 2288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:16.375519 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:16.375693 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:16.375724 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 18:43:16.375810 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 18:43:16.375942 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1521,"mutex_wait_us":3,"thread_start_us":1374,"threads_started":1}
I20260620 18:43:16.376468 11882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:16.366624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38712 (local address 127.23.115.3:43165)
0620 18:43:16.367595 (+ 971us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:16.367604 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:16.368141 (+ 537us) server_negotiation.cc:407] Connection header received
0620 18:43:16.368521 (+ 380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:16.368528 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:16.368719 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:16.368874 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:43:16.369857 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:16.371056 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:16.373438 (+ 2382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:16.374978 (+ 1540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:16.375093 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:16.375343 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:16.375875 (+ 532us) server_negotiation.cc:299] Negotiation successful
0620 18:43:16.376029 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":796,"thread_start_us":619,"threads_started":1}
I20260620 18:43:16.379225 11670 tablet_service.cc:1467] Processing CreateTablet for tablet 51e01956616b4cca896d5d474f07aae2 (DEFAULT_TABLE table=test-table [id=2c6de005bf934ddc82d0a256ce1fea69]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:43:16.379621 11800 tablet_service.cc:1467] Processing CreateTablet for tablet 51e01956616b4cca896d5d474f07aae2 (DEFAULT_TABLE table=test-table [id=2c6de005bf934ddc82d0a256ce1fea69]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:43:16.379590 11539 tablet_service.cc:1467] Processing CreateTablet for tablet 51e01956616b4cca896d5d474f07aae2 (DEFAULT_TABLE table=test-table [id=2c6de005bf934ddc82d0a256ce1fea69]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:43:16.380669 11670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51e01956616b4cca896d5d474f07aae2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:43:16.380874 11800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51e01956616b4cca896d5d474f07aae2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:43:16.380878 11539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51e01956616b4cca896d5d474f07aae2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:43:16.397154 11885 tablet_bootstrap.cc:492] T 51e01956616b4cca896d5d474f07aae2 P d118d82c6e6a4e74a69875cb4c85942c: Bootstrap starting.
I20260620 18:43:16.398021 11883 tablet_bootstrap.cc:492] T 51e01956616b4cca896d5d474f07aae2 P ede04a84739f4cad879be7903a016d55: Bootstrap starting.
I20260620 18:43:16.397607 11884 tablet_bootstrap.cc:492] T 51e01956616b4cca896d5d474f07aae2 P a4e73df13dd345f78693eca04fba7080: Bootstrap starting.
I20260620 18:43:16.401077 11885 tablet_bootstrap.cc:654] T 51e01956616b4cca896d5d474f07aae2 P d118d82c6e6a4e74a69875cb4c85942c: Neither blocks nor log segments found. Creating new log.
I20260620 18:43:16.401415 11883 tablet_bootstrap.cc:654] T 51e01956616b4cca896d5d474f07aae2 P ede04a84739f4cad879be7903a016d55: Neither blocks nor log segments found. Creating new log.
I20260620 18:43:16.402488 11885 log.cc:824] T 51e01956616b4cca896d5d474f07aae2 P d118d82c6e6a4e74a69875cb4c85942c: Log is configured to *not* fsync() on all Append() calls
I20260620 18:43:16.402594 11883 log.cc:824] T 51e01956616b4cca896d5d474f07aae2 P ede04a84739f4cad879be7903a016d55: Log is configured to *not* fsync() on all Append() calls
I20260620 18:43:16.402797 11884 tablet_bootstrap.cc:654] T 51e01956616b4cca896d5d474f07aae2 P a4e73df13dd345f78693eca04fba7080: Neither blocks nor log segments found. Creating new log.
I20260620 18:43:16.404104 11884 log.cc:824] T 51e01956616b4cca896d5d474f07aae2 P a4e73df13dd345f78693eca04fba7080: Log is configured to *not* fsync() on all Append() calls
I20260620 18:43:16.405755 11883 tablet_bootstrap.cc:492] T 51e01956616b4cca896d5d474f07aae2 P ede04a84739f4cad879be7903a016d55: No bootstrap required, opened a new log
I20260620 18:43:16.405751 11885 tablet_bootstrap.cc:492] T 51e01956616b4cca896d5d474f07aae2 P d118d82c6e6a4e74a69875cb4c85942c: No bootstrap required, opened a new log
I20260620 18:43:16.406040 11885 ts_tablet_manager.cc:1397] T 51e01956616b4cca896d5d474f07aae2 P d118d82c6e6a4e74a69875cb4c85942c: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.002s
I20260620 18:43:16.406041 11883 ts_tablet_manager.cc:1397] T 51e01956616b4cca896d5d474f07aae2 P ede04a84739f4cad879be7903a016d55: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260620 18:43:16.406378 11884 tablet_bootstrap.cc:492] T 51e01956616b4cca896d5d474f07aae2 P a4e73df13dd345f78693eca04fba7080: No bootstrap required, opened a new log
I20260620 18:43:16.406596 11884 ts_tablet_manager.cc:1397] T 51e01956616b4cca896d5d474f07aae2 P a4e73df13dd345f78693eca04fba7080: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260620 18:43:16.407361 11604 heartbeater.cc:503] Master 127.23.115.62:40877 was elected leader, sending a full tablet report...
I20260620 18:43:16.415864 11884 raft_consensus.cc:348] T 51e01956616b4cca896d5d474f07aae2 P a4e73df13dd345f78693eca04fba7080 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d118d82c6e6a4e74a69875cb4c85942c" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 44665 } } peers { permanent_uuid: "ede04a84739f4cad879be7903a016d55" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 39531 } } peers { permanent_uuid: "a4e73df13dd345f78693eca04fba7080" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 43165 } }
I20260620 18:43:16.416249 11884 raft_consensus.cc:374] T 51e01956616b4cca896d5d474f07aae2 P a4e73df13dd345f78693eca04fba7080 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:43:16.416333 11884 raft_consensus.cc:729] T 51e01956616b4cca896d5d474f07aae2 P a4e73df13dd345f78693eca04fba7080 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4e73df13dd345f78693eca04fba7080, State: Initialized, Role: FOLLOWER
I20260620 18:43:16.416662 11883 raft_consensus.cc:348] T 51e01956616b4cca896d5d474f07aae2 P ede04a84739f4cad879be7903a016d55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d118d82c6e6a4e74a69875cb4c85942c" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 44665 } } peers { permanent_uuid: "ede04a84739f4cad879be7903a016d55" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 39531 } } peers { permanent_uuid: "a4e73df13dd345f78693eca04fba7080" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 43165 } }
I20260620 18:43:16.416900 11884 consensus_queue.cc:260] T 51e01956616b4cca896d5d474f07aae2 P a4e73df13dd345f78693eca04fba7080 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d118d82c6e6a4e74a69875cb4c85942c" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 44665 } } peers { permanent_uuid: "ede04a84739f4cad879be7903a016d55" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 39531 } } peers { permanent_uuid: "a4e73df13dd345f78693eca04fba7080" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 43165 } }
I20260620 18:43:16.417059 11883 raft_consensus.cc:374] T 51e01956616b4cca896d5d474f07aae2 P ede04a84739f4cad879be7903a016d55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:43:16.417146 11883 raft_consensus.cc:729] T 51e01956616b4cca896d5d474f07aae2 P ede04a84739f4cad879be7903a016d55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ede04a84739f4cad879be7903a016d55, State: Initialized, Role: FOLLOWER
I20260620 18:43:16.417524 11885 raft_consensus.cc:348] T 51e01956616b4cca896d5d474f07aae2 P d118d82c6e6a4e74a69875cb4c85942c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d118d82c6e6a4e74a69875cb4c85942c" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 44665 } } peers { permanent_uuid: "ede04a84739f4cad879be7903a016d55" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 39531 } } peers { permanent_uuid: "a4e73df13dd345f78693eca04fba7080" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 43165 } }
I20260620 18:43:16.417936 11885 raft_consensus.cc:374] T 51e01956616b4cca896d5d474f07aae2 P d118d82c6e6a4e74a69875cb4c85942c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:43:16.417779 11883 consensus_queue.cc:260] T 51e01956616b4cca896d5d474f07aae2 P ede04a84739f4cad879be7903a016d55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d118d82c6e6a4e74a69875cb4c85942c" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 44665 } } peers { permanent_uuid: "ede04a84739f4cad879be7903a016d55" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 39531 } } peers { permanent_uuid: "a4e73df13dd345f78693eca04fba7080" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 43165 } }
I20260620 18:43:16.417999 11865 heartbeater.cc:503] Master 127.23.115.62:40877 was elected leader, sending a full tablet report...
I20260620 18:43:16.418020 11885 raft_consensus.cc:729] T 51e01956616b4cca896d5d474f07aae2 P d118d82c6e6a4e74a69875cb4c85942c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d118d82c6e6a4e74a69875cb4c85942c, State: Initialized, Role: FOLLOWER
I20260620 18:43:16.418691 11884 ts_tablet_manager.cc:1428] T 51e01956616b4cca896d5d474f07aae2 P a4e73df13dd345f78693eca04fba7080: Time spent starting tablet: real 0.012s user 0.003s sys 0.009s
I20260620 18:43:16.418584 11885 consensus_queue.cc:260] T 51e01956616b4cca896d5d474f07aae2 P d118d82c6e6a4e74a69875cb4c85942c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d118d82c6e6a4e74a69875cb4c85942c" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 44665 } } peers { permanent_uuid: "ede04a84739f4cad879be7903a016d55" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 39531 } } peers { permanent_uuid: "a4e73df13dd345f78693eca04fba7080" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 43165 } }
I20260620 18:43:16.420398 11885 ts_tablet_manager.cc:1428] T 51e01956616b4cca896d5d474f07aae2 P d118d82c6e6a4e74a69875cb4c85942c: Time spent starting tablet: real 0.014s user 0.010s sys 0.005s
I20260620 18:43:16.424085 11735 heartbeater.cc:503] Master 127.23.115.62:40877 was elected leader, sending a full tablet report...
I20260620 18:43:16.424237 11883 ts_tablet_manager.cc:1428] T 51e01956616b4cca896d5d474f07aae2 P ede04a84739f4cad879be7903a016d55: Time spent starting tablet: real 0.018s user 0.018s sys 0.001s
I20260620 18:43:16.477888 11891 raft_consensus.cc:482] T 51e01956616b4cca896d5d474f07aae2 P d118d82c6e6a4e74a69875cb4c85942c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:43:16.478458 11891 raft_consensus.cc:504] T 51e01956616b4cca896d5d474f07aae2 P d118d82c6e6a4e74a69875cb4c85942c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d118d82c6e6a4e74a69875cb4c85942c" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 44665 } } peers { permanent_uuid: "ede04a84739f4cad879be7903a016d55" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 39531 } } peers { permanent_uuid: "a4e73df13dd345f78693eca04fba7080" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 43165 } }
I20260620 18:43:16.480190 11891 leader_election.cc:288] T 51e01956616b4cca896d5d474f07aae2 P d118d82c6e6a4e74a69875cb4c85942c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ede04a84739f4cad879be7903a016d55 (127.23.115.2:39531), a4e73df13dd345f78693eca04fba7080 (127.23.115.3:43165)
I20260620 18:43:16.485319 11890 raft_consensus.cc:482] T 51e01956616b4cca896d5d474f07aae2 P ede04a84739f4cad879be7903a016d55 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:43:16.485631 11890 raft_consensus.cc:504] T 51e01956616b4cca896d5d474f07aae2 P ede04a84739f4cad879be7903a016d55 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d118d82c6e6a4e74a69875cb4c85942c" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 44665 } } peers { permanent_uuid: "ede04a84739f4cad879be7903a016d55" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 39531 } } peers { permanent_uuid: "a4e73df13dd345f78693eca04fba7080" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 43165 } }
I20260620 18:43:16.487648 11882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:16.480815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.1:42155 (local address 127.23.115.3:43165)
0620 18:43:16.481264 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:16.481271 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:16.482075 (+ 804us) server_negotiation.cc:407] Connection header received
0620 18:43:16.482245 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:16.482249 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:16.482310 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:16.482385 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:43:16.483000 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:16.483691 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:16.485854 (+ 2163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:16.486726 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:16.486904 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:16.487004 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:16.487256 (+ 252us) server_negotiation.cc:299] Negotiation successful
0620 18:43:16.487406 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 18:43:16.488030 11892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:16.480573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.2:39531 (local address 127.23.115.1:59873)
0620 18:43:16.481922 (+ 1349us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:16.481941 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:16.482087 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:16.482410 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:16.482414 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:16.482458 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:43:16.482886 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:16.482896 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:16.483825 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:16.483829 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:16.485677 (+ 1848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:16.485690 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:16.487264 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:16.487270 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:16.487423 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:16.487452 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 18:43:16.487537 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 18:43:16.487653 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1215,"mutex_wait_us":169,"spinlock_wait_cycles":2816,"thread_start_us":955,"threads_started":1}
I20260620 18:43:16.488063 11880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:16.480717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.1:59873 (local address 127.23.115.2:39531)
0620 18:43:16.481275 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:16.481280 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:16.482031 (+ 751us) server_negotiation.cc:407] Connection header received
0620 18:43:16.482225 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:16.482230 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:16.482299 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:16.482382 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:43:16.483024 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:16.483693 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:16.485918 (+ 2225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:16.487084 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:16.487232 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:16.487295 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:16.487575 (+ 280us) server_negotiation.cc:299] Negotiation successful
0620 18:43:16.487696 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159}
I20260620 18:43:16.488535 11890 leader_election.cc:288] T 51e01956616b4cca896d5d474f07aae2 P ede04a84739f4cad879be7903a016d55 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d118d82c6e6a4e74a69875cb4c85942c (127.23.115.1:44665), a4e73df13dd345f78693eca04fba7080 (127.23.115.3:43165)
I20260620 18:43:16.489161 11893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:16.480672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.3:43165 (local address 127.23.115.1:42155)
0620 18:43:16.481970 (+ 1298us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:16.481988 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:16.482110 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:16.482410 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:16.482414 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:16.482449 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:43:16.482812 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:16.482820 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:16.483809 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:16.483814 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:16.485647 (+ 1833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:16.485662 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:16.486904 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:16.486910 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:16.487085 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:16.487108 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 18:43:16.487980 (+ 872us) client_negotiation.cc:240] Negotiation successful
0620 18:43:16.488372 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1179,"mutex_wait_us":97,"spinlock_wait_cycles":3328,"thread_start_us":994,"threads_started":1}
I20260620 18:43:16.488983 11690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51e01956616b4cca896d5d474f07aae2" candidate_uuid: "d118d82c6e6a4e74a69875cb4c85942c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ede04a84739f4cad879be7903a016d55" is_pre_election: true
I20260620 18:43:16.489498 11690 raft_consensus.cc:2454] T 51e01956616b4cca896d5d474f07aae2 P ede04a84739f4cad879be7903a016d55 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d118d82c6e6a4e74a69875cb4c85942c in term 0.
I20260620 18:43:16.490449 11493 leader_election.cc:302] T 51e01956616b4cca896d5d474f07aae2 P d118d82c6e6a4e74a69875cb4c85942c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d118d82c6e6a4e74a69875cb4c85942c, ede04a84739f4cad879be7903a016d55; no voters:
I20260620 18:43:16.490574 11820 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51e01956616b4cca896d5d474f07aae2" candidate_uuid: "d118d82c6e6a4e74a69875cb4c85942c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4e73df13dd345f78693eca04fba7080" is_pre_election: true
I20260620 18:43:16.490897 11891 raft_consensus.cc:2781] T 51e01956616b4cca896d5d474f07aae2 P d118d82c6e6a4e74a69875cb4c85942c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:43:16.491067 11891 raft_consensus.cc:482] T 51e01956616b4cca896d5d474f07aae2 P d118d82c6e6a4e74a69875cb4c85942c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:43:16.491168 11891 raft_consensus.cc:3037] T 51e01956616b4cca896d5d474f07aae2 P d118d82c6e6a4e74a69875cb4c85942c [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:43:16.491153 11820 raft_consensus.cc:2454] T 51e01956616b4cca896d5d474f07aae2 P a4e73df13dd345f78693eca04fba7080 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d118d82c6e6a4e74a69875cb4c85942c in term 0.
I20260620 18:43:16.494554 11891 raft_consensus.cc:504] T 51e01956616b4cca896d5d474f07aae2 P d118d82c6e6a4e74a69875cb4c85942c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d118d82c6e6a4e74a69875cb4c85942c" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 44665 } } peers { permanent_uuid: "ede04a84739f4cad879be7903a016d55" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 39531 } } peers { permanent_uuid: "a4e73df13dd345f78693eca04fba7080" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 43165 } }
I20260620 18:43:16.495494 11879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:16.488468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.2:42681 (local address 127.23.115.1:44665)
0620 18:43:16.488631 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:16.488635 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:16.489293 (+ 658us) server_negotiation.cc:407] Connection header received
0620 18:43:16.489505 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:16.489510 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:16.489581 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:16.489647 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:43:16.490905 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:16.491944 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:16.493836 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:16.494587 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:16.494715 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:16.494792 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:16.495110 (+ 318us) server_negotiation.cc:299] Negotiation successful
0620 18:43:16.495229 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 18:43:16.495498 11894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:16.487942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.1:44665 (local address 127.23.115.2:42681)
0620 18:43:16.489199 (+ 1257us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:16.489223 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:16.489389 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:16.490232 (+ 843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:16.490238 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:16.490328 (+ 90us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:43:16.490723 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:16.490736 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:16.492110 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:16.492113 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:16.493650 (+ 1537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:16.493661 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:16.494768 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:16.494774 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:16.494910 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:16.494967 (+ 57us) client_negotiation.cc:769] Sending connection context
0620 18:43:16.495058 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 18:43:16.495184 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1116,"thread_start_us":876,"threads_started":1}
I20260620 18:43:16.496534 11690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51e01956616b4cca896d5d474f07aae2" candidate_uuid: "d118d82c6e6a4e74a69875cb4c85942c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ede04a84739f4cad879be7903a016d55"
I20260620 18:43:16.496755 11690 raft_consensus.cc:3037] T 51e01956616b4cca896d5d474f07aae2 P ede04a84739f4cad879be7903a016d55 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:43:16.499748 11891 leader_election.cc:288] T 51e01956616b4cca896d5d474f07aae2 P d118d82c6e6a4e74a69875cb4c85942c [CANDIDATE]: Term 1 election: Requested vote from peers ede04a84739f4cad879be7903a016d55 (127.23.115.2:39531), a4e73df13dd345f78693eca04fba7080 (127.23.115.3:43165)
I20260620 18:43:16.500037 11559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51e01956616b4cca896d5d474f07aae2" candidate_uuid: "ede04a84739f4cad879be7903a016d55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d118d82c6e6a4e74a69875cb4c85942c" is_pre_election: true
I20260620 18:43:16.500520 11559 raft_consensus.cc:2379] T 51e01956616b4cca896d5d474f07aae2 P d118d82c6e6a4e74a69875cb4c85942c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ede04a84739f4cad879be7903a016d55 in current term 1: Already voted for candidate d118d82c6e6a4e74a69875cb4c85942c in this term.
I20260620 18:43:16.500536 11690 raft_consensus.cc:2454] T 51e01956616b4cca896d5d474f07aae2 P ede04a84739f4cad879be7903a016d55 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d118d82c6e6a4e74a69875cb4c85942c in term 1.
I20260620 18:43:16.500540 11820 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51e01956616b4cca896d5d474f07aae2" candidate_uuid: "d118d82c6e6a4e74a69875cb4c85942c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4e73df13dd345f78693eca04fba7080"
I20260620 18:43:16.500730 11820 raft_consensus.cc:3037] T 51e01956616b4cca896d5d474f07aae2 P a4e73df13dd345f78693eca04fba7080 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:43:16.500942 11493 leader_election.cc:302] T 51e01956616b4cca896d5d474f07aae2 P d118d82c6e6a4e74a69875cb4c85942c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d118d82c6e6a4e74a69875cb4c85942c, ede04a84739f4cad879be7903a016d55; no voters:
I20260620 18:43:16.501313 11891 raft_consensus.cc:2781] T 51e01956616b4cca896d5d474f07aae2 P d118d82c6e6a4e74a69875cb4c85942c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:43:16.501490 11882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:16.489834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.115.2:45211 (local address 127.23.115.3:43165)
0620 18:43:16.490155 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:16.490160 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:16.495796 (+ 5636us) server_negotiation.cc:407] Connection header received
0620 18:43:16.495955 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:16.495961 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:16.496046 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:16.496119 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:43:16.496660 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:16.497556 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:16.499263 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:16.500388 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:16.500553 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:16.500620 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:16.501094 (+ 474us) server_negotiation.cc:299] Negotiation successful
0620 18:43:16.501225 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201}
I20260620 18:43:16.501825 11891 raft_consensus.cc:686] T 51e01956616b4cca896d5d474f07aae2 P d118d82c6e6a4e74a69875cb4c85942c [term 1 LEADER]: Becoming Leader. State: Replica: d118d82c6e6a4e74a69875cb4c85942c, State: Running, Role: LEADER
I20260620 18:43:16.502151 11894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:16.489272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.3:43165 (local address 127.23.115.2:45211)
0620 18:43:16.495703 (+ 6431us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:16.495718 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:16.495819 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:16.496313 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:16.496317 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:16.496342 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:43:16.496486 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:16.496497 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:16.497686 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:16.497691 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:16.499118 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:16.499130 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:16.500710 (+ 1580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:16.500716 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:16.500908 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:16.500942 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 18:43:16.501653 (+ 711us) client_negotiation.cc:240] Negotiation successful
0620 18:43:16.501810 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6316,"thread_start_us":9849,"threads_started":1}
I20260620 18:43:16.502341 11891 consensus_queue.cc:237] T 51e01956616b4cca896d5d474f07aae2 P d118d82c6e6a4e74a69875cb4c85942c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d118d82c6e6a4e74a69875cb4c85942c" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 44665 } } peers { permanent_uuid: "ede04a84739f4cad879be7903a016d55" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 39531 } } peers { permanent_uuid: "a4e73df13dd345f78693eca04fba7080" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 43165 } }
I20260620 18:43:16.502782 11819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51e01956616b4cca896d5d474f07aae2" candidate_uuid: "ede04a84739f4cad879be7903a016d55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4e73df13dd345f78693eca04fba7080" is_pre_election: true
I20260620 18:43:16.503650 11625 leader_election.cc:302] T 51e01956616b4cca896d5d474f07aae2 P ede04a84739f4cad879be7903a016d55 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ede04a84739f4cad879be7903a016d55; no voters: a4e73df13dd345f78693eca04fba7080, d118d82c6e6a4e74a69875cb4c85942c
I20260620 18:43:16.503994 11890 raft_consensus.cc:2726] T 51e01956616b4cca896d5d474f07aae2 P ede04a84739f4cad879be7903a016d55 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 18:43:16.504370 11820 raft_consensus.cc:2454] T 51e01956616b4cca896d5d474f07aae2 P a4e73df13dd345f78693eca04fba7080 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d118d82c6e6a4e74a69875cb4c85942c in term 1.
I20260620 18:43:16.507495 11417 catalog_manager.cc:5495] T 51e01956616b4cca896d5d474f07aae2 P d118d82c6e6a4e74a69875cb4c85942c reported cstate change: term changed from 0 to 1, leader changed from <none> to d118d82c6e6a4e74a69875cb4c85942c (127.23.115.1). New cstate: current_term: 1 leader_uuid: "d118d82c6e6a4e74a69875cb4c85942c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d118d82c6e6a4e74a69875cb4c85942c" member_type: VOTER last_known_addr { host: "127.23.115.1" port: 44665 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ede04a84739f4cad879be7903a016d55" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 39531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4e73df13dd345f78693eca04fba7080" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 43165 } health_report { overall_health: UNKNOWN } } }
I20260620 18:43:16.541824 11879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:16.536408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54364 (local address 127.23.115.1:44665)
0620 18:43:16.536565 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:16.536570 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:16.536617 (+ 47us) server_negotiation.cc:407] Connection header received
0620 18:43:16.536787 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:16.536791 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:16.536902 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:16.537127 (+ 225us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:43:16.537524 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:16.538305 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:16.539764 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:16.540168 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:16.540282 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:16.540720 (+ 438us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:43:16.541230 (+ 510us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:43:16.541359 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:16.541469 (+ 110us) server_negotiation.cc:299] Negotiation successful
0620 18:43:16.541547 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260620 18:43:16.549736 11866 tablet.cc:2369] T 51e01956616b4cca896d5d474f07aae2 P a4e73df13dd345f78693eca04fba7080: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:43:16.549925 11690 raft_consensus.cc:1261] T 51e01956616b4cca896d5d474f07aae2 P ede04a84739f4cad879be7903a016d55 [term 1 FOLLOWER]: Refusing update from remote peer d118d82c6e6a4e74a69875cb4c85942c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:43:16.549964 11820 raft_consensus.cc:1261] T 51e01956616b4cca896d5d474f07aae2 P a4e73df13dd345f78693eca04fba7080 [term 1 FOLLOWER]: Refusing update from remote peer d118d82c6e6a4e74a69875cb4c85942c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:43:16.550820 11896 consensus_queue.cc:1035] T 51e01956616b4cca896d5d474f07aae2 P d118d82c6e6a4e74a69875cb4c85942c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ede04a84739f4cad879be7903a016d55" member_type: VOTER last_known_addr { host: "127.23.115.2" port: 39531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:43:16.551200 11891 consensus_queue.cc:1035] T 51e01956616b4cca896d5d474f07aae2 P d118d82c6e6a4e74a69875cb4c85942c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4e73df13dd345f78693eca04fba7080" member_type: VOTER last_known_addr { host: "127.23.115.3" port: 43165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:43:16.562186 11899 mvcc.cc:205] Tried to move back new op lower bound from 7298994161859178496 to 7298994161678155776. Current Snapshot: MvccSnapshot[applied={T|T < 7298994161859178496}]
I20260620 18:43:16.573856 11417 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44642:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:43:16.574112 11417 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44642:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:43:16.577798 11417 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a3c7620f7e0a4afb9a6371cc242a4b09: Sending DeleteTablet for 3 replicas of tablet 51e01956616b4cca896d5d474f07aae2
I20260620 18:43:16.578922 11670 tablet_service.cc:1514] Processing DeleteTablet for tablet 51e01956616b4cca896d5d474f07aae2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:43:16 UTC) from {username='slave'} at 127.0.0.1:44280
I20260620 18:43:16.579007 11539 tablet_service.cc:1514] Processing DeleteTablet for tablet 51e01956616b4cca896d5d474f07aae2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:43:16 UTC) from {username='slave'} at 127.0.0.1:54352
I20260620 18:43:16.579447 11800 tablet_service.cc:1514] Processing DeleteTablet for tablet 51e01956616b4cca896d5d474f07aae2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:43:16 UTC) from {username='slave'} at 127.0.0.1:38712
I20260620 18:43:16.580036 11909 tablet_replica.cc:330] T 51e01956616b4cca896d5d474f07aae2 P d118d82c6e6a4e74a69875cb4c85942c: stopping tablet replica
I20260620 18:43:16.580446 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 11477
I20260620 18:43:16.580514 11909 raft_consensus.cc:2229] T 51e01956616b4cca896d5d474f07aae2 P d118d82c6e6a4e74a69875cb4c85942c [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:43:16.581396 11909 raft_consensus.cc:2258] T 51e01956616b4cca896d5d474f07aae2 P d118d82c6e6a4e74a69875cb4c85942c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:43:16.587240 11909 ts_tablet_manager.cc:1905] T 51e01956616b4cca896d5d474f07aae2 P d118d82c6e6a4e74a69875cb4c85942c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:43:16.590615 11910 tablet_replica.cc:330] T 51e01956616b4cca896d5d474f07aae2 P a4e73df13dd345f78693eca04fba7080: stopping tablet replica
I20260620 18:43:16.591109 11910 raft_consensus.cc:2229] T 51e01956616b4cca896d5d474f07aae2 P a4e73df13dd345f78693eca04fba7080 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:43:16.591602 11910 raft_consensus.cc:2258] T 51e01956616b4cca896d5d474f07aae2 P a4e73df13dd345f78693eca04fba7080 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:43:16.592348 11879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:16.582288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54366 (local address 127.23.115.1:44665)
0620 18:43:16.582625 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:16.582631 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:16.582655 (+ 24us) server_negotiation.cc:407] Connection header received
0620 18:43:16.582859 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:16.582866 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:16.583293 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:16.583522 (+ 229us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:16.584641 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:16.585781 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:16.587822 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:16.588540 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:16.588673 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:16.590245 (+ 1572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:16.590258 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:16.590275 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:43:16.591154 (+ 879us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:16.591245 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:43:16.591273 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:16.591361 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:16.591833 (+ 472us) server_negotiation.cc:299] Negotiation successful
0620 18:43:16.591976 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204}
I20260620 18:43:16.592422 11908 tablet_replica.cc:330] T 51e01956616b4cca896d5d474f07aae2 P ede04a84739f4cad879be7903a016d55: stopping tablet replica
I20260620 18:43:16.592792 11908 raft_consensus.cc:2229] T 51e01956616b4cca896d5d474f07aae2 P ede04a84739f4cad879be7903a016d55 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:43:16.593505 11908 raft_consensus.cc:2258] T 51e01956616b4cca896d5d474f07aae2 P ede04a84739f4cad879be7903a016d55 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:43:16.593760 11599 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:43:16.596303 11908 ts_tablet_manager.cc:1905] T 51e01956616b4cca896d5d474f07aae2 P ede04a84739f4cad879be7903a016d55: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:43:16.596303 11910 ts_tablet_manager.cc:1905] T 51e01956616b4cca896d5d474f07aae2 P a4e73df13dd345f78693eca04fba7080: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:43:16.604590 11910 ts_tablet_manager.cc:1918] T 51e01956616b4cca896d5d474f07aae2 P a4e73df13dd345f78693eca04fba7080: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:43:16.604591 11908 ts_tablet_manager.cc:1918] T 51e01956616b4cca896d5d474f07aae2 P ede04a84739f4cad879be7903a016d55: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:43:16.604800 11908 log.cc:1196] T 51e01956616b4cca896d5d474f07aae2 P ede04a84739f4cad879be7903a016d55: Deleting WAL directory at /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980676253068-24012-0/minicluster-data/ts-1/wal/wals/51e01956616b4cca896d5d474f07aae2
I20260620 18:43:16.604799 11910 log.cc:1196] T 51e01956616b4cca896d5d474f07aae2 P a4e73df13dd345f78693eca04fba7080: Deleting WAL directory at /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980676253068-24012-0/minicluster-data/ts-2/wal/wals/51e01956616b4cca896d5d474f07aae2
I20260620 18:43:16.605342 11910 ts_tablet_manager.cc:1939] T 51e01956616b4cca896d5d474f07aae2 P a4e73df13dd345f78693eca04fba7080: Deleting consensus metadata
I20260620 18:43:16.605342 11908 ts_tablet_manager.cc:1939] T 51e01956616b4cca896d5d474f07aae2 P ede04a84739f4cad879be7903a016d55: Deleting consensus metadata
I20260620 18:43:16.607473 11404 catalog_manager.cc:4841] TS a4e73df13dd345f78693eca04fba7080 (127.23.115.3:43165): tablet 51e01956616b4cca896d5d474f07aae2 (table test-table [id=2c6de005bf934ddc82d0a256ce1fea69]) successfully deleted
I20260620 18:43:16.607429 11403 catalog_manager.cc:4841] TS ede04a84739f4cad879be7903a016d55 (127.23.115.2:39531): tablet 51e01956616b4cca896d5d474f07aae2 (table test-table [id=2c6de005bf934ddc82d0a256ce1fea69]) successfully deleted
I20260620 18:43:16.691257 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 11477
W20260620 18:43:16.707855 11404 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.115.1:44665 (error 108)
I20260620 18:43:16.708105 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 11608
I20260620 18:43:16.710685 11881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:16.710162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.1:44665 (local address 127.0.0.1:54380)
0620 18:43:16.710341 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:16.710475 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.1:44665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260620 18:43:16.711094 11404 catalog_manager.cc:4584] TS d118d82c6e6a4e74a69875cb4c85942c (127.23.115.1:44665): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 51e01956616b4cca896d5d474f07aae2: Network error: Client connection negotiation failed: client connection to 127.23.115.1:44665: connect: Connection refused (error 111)
I20260620 18:43:16.716657 11880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:16.709468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44288 (local address 127.23.115.2:39531)
0620 18:43:16.709633 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:16.709639 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:16.709657 (+ 18us) server_negotiation.cc:407] Connection header received
0620 18:43:16.709721 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:16.709726 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:16.709874 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:16.709985 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:16.710544 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:16.711649 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:16.713509 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:16.714103 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:16.714203 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:16.715326 (+ 1123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:16.715341 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:16.715356 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:43:16.715926 (+ 570us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:16.715984 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:43:16.716008 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:16.716102 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:16.716237 (+ 135us) server_negotiation.cc:299] Negotiation successful
0620 18:43:16.716326 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 18:43:16.717372 11730 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:43:16.764887 11881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:16.764268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.1:44665 (local address 127.0.0.1:54384)
0620 18:43:16.764520 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:16.764644 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.1:44665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260620 18:43:16.766016 11881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:16.765621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.1:44665 (local address 127.0.0.1:54390)
0620 18:43:16.765810 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:16.765880 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.1:44665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260620 18:43:16.806975 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 11608
I20260620 18:43:16.820933 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 11738
I20260620 18:43:16.826543 11881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:16.826101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.1:44665 (local address 127.0.0.1:54402)
0620 18:43:16.826300 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:16.826374 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.1:44665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260620 18:43:16.827508 11881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:16.827106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.1:44665 (local address 127.0.0.1:54412)
0620 18:43:16.827285 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:16.827376 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.1:44665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260620 18:43:16.827580 11882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:16.821738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38724 (local address 127.23.115.3:43165)
0620 18:43:16.821900 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:16.821905 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:16.821962 (+ 57us) server_negotiation.cc:407] Connection header received
0620 18:43:16.822188 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:16.822193 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:16.822296 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:16.822381 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:16.822752 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:16.823557 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:16.824748 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:16.825121 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:16.825211 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:16.826195 (+ 984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:16.826203 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:16.826218 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:43:16.826845 (+ 627us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:16.826893 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:43:16.826919 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:16.827041 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:16.827186 (+ 145us) server_negotiation.cc:299] Negotiation successful
0620 18:43:16.827283 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 18:43:16.828114 11860 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:43:16.908356 11881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:16.907850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.1:44665 (local address 127.0.0.1:54426)
0620 18:43:16.908095 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:16.908192 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.1:44665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260620 18:43:16.909385 11881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:16.909009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.1:44665 (local address 127.0.0.1:54428)
0620 18:43:16.909196 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:16.909266 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.1:44665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260620 18:43:16.930083 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 11738
I20260620 18:43:16.944360 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 11387
I20260620 18:43:16.945854 11448 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:43:17.053365 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 11387
2026-06-20T18:43:17Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2694 ms)
[----------] 6 tests from EncryptionPolicyTest (21163 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980676253068-24012-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:43:17 dist-test-slave-d645 krb5kdc[11919](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:43:17 dist-test-slave-d645 krb5kdc[11919](info): set up 2 sockets
Jun 20 18:43:17 dist-test-slave-d645 krb5kdc[11919](info): commencing operation
krb5kdc: starting...
W20260620 18:43:18.083511 24012 mini_kdc.cc:121] Time spent starting KDC: real 0.992s user 0.007s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980676253068-24012-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980676253068-24012-0/krb5kdc/test-user.keytab.
Jun 20 18:43:18 dist-test-slave-d645 krb5kdc[11919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980998, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:43:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:43:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980676253068-24012-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980676253068-24012-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 18:43:18.332926 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980676253068-24012-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980676253068-24012-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980676253068-24012-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40813
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980676253068-24012-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40813
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:43:18.649204 11932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:43:18.649505 11932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:43:18.649624 11932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:43:18.657286 11932 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:43:18.657423 11932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:43:18.657480 11932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:43:18.657564 11932 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:43:18.657625 11932 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:43:18.667956 11932 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980676253068-24012-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980676253068-24012-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40813
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980676253068-24012-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980676253068-24012-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:43:18.669210 11932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:43:18.670445 11932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:43:18.681419 11938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:18.681636 11940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:18.681727 11937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:18.682204 11932 server_base.cc:1034] running on GCE node
I20260620 18:43:18.682924 11932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:43:18.684298 11932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:43:18.685508 11932 hybrid_clock.cc:648] HybridClock initialized: now 1781980998685460 us; error 37 us; skew 500 ppm
I20260620 18:43:18.685828 11932 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:43:18 dist-test-slave-d645 krb5kdc[11919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980998, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:43:18.688740 11932 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 18:43:18.690095 11932 webserver.cc:460] Webserver started at http://127.0.0.1:33119/ using document root <none> and password file <none>
I20260620 18:43:18.690712 11932 fs_manager.cc:359] Metadata directory not provided
I20260620 18:43:18.690811 11932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:43:18.691093 11932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:43:18.694125 11932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980676253068-24012-0/minicluster-data/master-0/data/instance:
uuid: "1a34999069a8470fb65a3e45ccde9c59"
format_stamp: "Formatted at 2026-06-20 18:43:18 on dist-test-slave-d645"
server_key: "f3c0344b44fcb27da5efc42d72d3c378"
server_key_iv: "799d49f4b4631249ef132064df1ec063"
server_key_version: "encryptionkey@0"
I20260620 18:43:18.694893 11932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980676253068-24012-0/minicluster-data/master-0/wal/instance:
uuid: "1a34999069a8470fb65a3e45ccde9c59"
format_stamp: "Formatted at 2026-06-20 18:43:18 on dist-test-slave-d645"
server_key: "f3c0344b44fcb27da5efc42d72d3c378"
server_key_iv: "799d49f4b4631249ef132064df1ec063"
server_key_version: "encryptionkey@0"
I20260620 18:43:18.700095 11932 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 18:43:18.703938 11947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:18.705639 11932 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:43:18.705823 11932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980676253068-24012-0/minicluster-data/master-0/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980676253068-24012-0/minicluster-data/master-0/wal
uuid: "1a34999069a8470fb65a3e45ccde9c59"
format_stamp: "Formatted at 2026-06-20 18:43:18 on dist-test-slave-d645"
server_key: "f3c0344b44fcb27da5efc42d72d3c378"
server_key_iv: "799d49f4b4631249ef132064df1ec063"
server_key_version: "encryptionkey@0"
I20260620 18:43:18.706032 11932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980676253068-24012-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980676253068-24012-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980676253068-24012-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:43:18.723080 11932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:43:18.727191 11932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:43:18.727447 11932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:43:18.743326 11932 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40813
I20260620 18:43:18.743360 11998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40813 every 8 connection(s)
I20260620 18:43:18.744585 11932 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
I20260620 18:43:18.745985 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 11932
I20260620 18:43:18.746191 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980676253068-24012-0/minicluster-data/master-0/wal/instance
I20260620 18:43:18.747059 24012 external_mini_cluster.cc:1428] Setting key d9ea1e616ed698578fc5ee0758f9e952
I20260620 18:43:18.749073 11999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 18:43:18 dist-test-slave-d645 krb5kdc[11919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980998, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 18:43:18.762323 11999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a34999069a8470fb65a3e45ccde9c59: Bootstrap starting.
I20260620 18:43:18.766230 11999 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a34999069a8470fb65a3e45ccde9c59: Neither blocks nor log segments found. Creating new log.
I20260620 18:43:18.767577 11999 log.cc:824] T 00000000000000000000000000000000 P 1a34999069a8470fb65a3e45ccde9c59: Log is configured to *not* fsync() on all Append() calls
I20260620 18:43:18.770877 11999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a34999069a8470fb65a3e45ccde9c59: No bootstrap required, opened a new log
I20260620 18:43:18.771880 12002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:18.748899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52607 (local address 127.0.0.1:40813)
0620 18:43:18.749623 (+ 724us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:18.749638 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:18.749683 (+ 45us) server_negotiation.cc:407] Connection header received
0620 18:43:18.751817 (+ 2134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:18.751844 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:18.752337 (+ 493us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:18.752755 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:18.754389 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:18.756398 (+ 2009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:18.757666 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:18.758671 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:18.758877 (+ 206us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:18.763160 (+ 4283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:18.763171 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:18.763188 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:18.763238 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:18.767190 (+ 3952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:18.768102 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:18.768112 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:18.768121 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:18.768325 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:18.768828 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:18.768831 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:18.768833 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:18.769488 (+ 655us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:18.769779 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:18.770222 (+ 443us) server_negotiation.cc:299] Negotiation successful
0620 18:43:18.770579 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":512,"thread_start_us":296,"threads_started":1}
I20260620 18:43:18.775875 24012 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:43:18.779237 11999 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a34999069a8470fb65a3e45ccde9c59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a34999069a8470fb65a3e45ccde9c59" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40813 } }
I20260620 18:43:18.779592 11999 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1a34999069a8470fb65a3e45ccde9c59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:43:18.779681 11999 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a34999069a8470fb65a3e45ccde9c59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a34999069a8470fb65a3e45ccde9c59, State: Initialized, Role: FOLLOWER
I20260620 18:43:18.780313 11999 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a34999069a8470fb65a3e45ccde9c59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a34999069a8470fb65a3e45ccde9c59" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40813 } }
I20260620 18:43:18.780601 11999 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1a34999069a8470fb65a3e45ccde9c59 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:43:18.780745 11999 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a34999069a8470fb65a3e45ccde9c59 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:43:18.780906 11999 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a34999069a8470fb65a3e45ccde9c59 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:43:18.783108 11999 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a34999069a8470fb65a3e45ccde9c59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a34999069a8470fb65a3e45ccde9c59" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40813 } }
I20260620 18:43:18.783603 11999 leader_election.cc:302] T 00000000000000000000000000000000 P 1a34999069a8470fb65a3e45ccde9c59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a34999069a8470fb65a3e45ccde9c59; no voters:
I20260620 18:43:18.784162 11999 leader_election.cc:288] T 00000000000000000000000000000000 P 1a34999069a8470fb65a3e45ccde9c59 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:43:18.784273 12009 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a34999069a8470fb65a3e45ccde9c59 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:43:18.786396 11999 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a34999069a8470fb65a3e45ccde9c59 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:43:18.787195 12009 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a34999069a8470fb65a3e45ccde9c59 [term 1 LEADER]: Becoming Leader. State: Replica: 1a34999069a8470fb65a3e45ccde9c59, State: Running, Role: LEADER
I20260620 18:43:18.787976 12009 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a34999069a8470fb65a3e45ccde9c59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a34999069a8470fb65a3e45ccde9c59" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40813 } }
I20260620 18:43:18.791355 12010 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a34999069a8470fb65a3e45ccde9c59 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1a34999069a8470fb65a3e45ccde9c59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a34999069a8470fb65a3e45ccde9c59" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40813 } } }
I20260620 18:43:18.791796 12010 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a34999069a8470fb65a3e45ccde9c59 [sys.catalog]: This master's current role is: LEADER
I20260620 18:43:18.792183 12011 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a34999069a8470fb65a3e45ccde9c59 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a34999069a8470fb65a3e45ccde9c59. Latest consensus state: current_term: 1 leader_uuid: "1a34999069a8470fb65a3e45ccde9c59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a34999069a8470fb65a3e45ccde9c59" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40813 } } }
I20260620 18:43:18.792454 12011 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a34999069a8470fb65a3e45ccde9c59 [sys.catalog]: This master's current role is: LEADER
I20260620 18:43:18.795770 12002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:18.780359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34853 (local address 127.0.0.1:40813)
0620 18:43:18.780515 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:18.780520 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:18.780982 (+ 462us) server_negotiation.cc:407] Connection header received
0620 18:43:18.781139 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:18.781148 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:18.781216 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:18.781302 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:18.782565 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:18.783465 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:18.784934 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:18.785450 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:18.785521 (+ 71us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:18.788384 (+ 2863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:18.788391 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:18.788395 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:18.788436 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:18.791892 (+ 3456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:18.792910 (+ 1018us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:18.792915 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:18.792932 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:18.793109 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:18.793681 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:18.793686 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:18.793689 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:18.794101 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:18.794406 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:18.795182 (+ 776us) server_negotiation.cc:299] Negotiation successful
0620 18:43:18.795390 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 18:43:18.804581 12016 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:43:18.808697 12016 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:43:18.816926 12016 catalog_manager.cc:1347] Generated new cluster ID: 609cec1a55b1493997a5bd8515bb2ad1
I20260620 18:43:18.817042 12016 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:43:18.832896 12016 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:43:18.834789 12016 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:43:18.845717 12016 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1a34999069a8470fb65a3e45ccde9c59: Generated new TSK 0
I20260620 18:43:18.846511 12016 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:43:18.860152 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 11932
I20260620 18:43:18.861486 11994 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:43:18.952601 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 11932
W20260620 18:43:18.967054 11917 connection.cc:535] client connection to 127.0.0.1:40813 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40813): Connection reset by peer (error 104)
2026-06-20T18:43:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1894 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980676253068-24012-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:43:19 dist-test-slave-d645 krb5kdc[12031](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:43:19 dist-test-slave-d645 krb5kdc[12031](info): set up 2 sockets
Jun 20 18:43:19 dist-test-slave-d645 krb5kdc[12031](info): commencing operation
krb5kdc: starting...
W20260620 18:43:19.971302 24012 mini_kdc.cc:121] Time spent starting KDC: real 0.986s user 0.004s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980676253068-24012-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980676253068-24012-0/krb5kdc/test-user.keytab.
Jun 20 18:43:20 dist-test-slave-d645 krb5kdc[12031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:43:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:43:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980676253068-24012-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980676253068-24012-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 18:43:20.212040 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980676253068-24012-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980676253068-24012-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980676253068-24012-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35489
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980676253068-24012-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35489
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:43:20.533782 12044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:43:20.534066 12044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:43:20.534174 12044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:43:20.541391 12044 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:43:20.541492 12044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:43:20.541566 12044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:43:20.541648 12044 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:43:20.541687 12044 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:43:20.551406 12044 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980676253068-24012-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980676253068-24012-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35489
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980676253068-24012-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980676253068-24012-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:43:20.552618 12044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:43:20.553818 12044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:43:20.564827 12050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:20.564872 12049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:20.564997 12052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:20.565595 12044 server_base.cc:1034] running on GCE node
I20260620 18:43:20.566337 12044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:43:20.567744 12044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:43:20.568971 12044 hybrid_clock.cc:648] HybridClock initialized: now 1781981000568909 us; error 44 us; skew 500 ppm
I20260620 18:43:20.569368 12044 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:43:20 dist-test-slave-d645 krb5kdc[12031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981000, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:43:20.572422 12044 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 18:43:20.573946 12044 webserver.cc:460] Webserver started at http://127.0.0.1:45827/ using document root <none> and password file <none>
I20260620 18:43:20.574589 12044 fs_manager.cc:359] Metadata directory not provided
I20260620 18:43:20.574694 12044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:43:20.575016 12044 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:43:20.577922 12044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980676253068-24012-0/minicluster-data/master-0/data/instance:
uuid: "e481c921702a47eb8a2989431c1ec9d6"
format_stamp: "Formatted at 2026-06-20 18:43:20 on dist-test-slave-d645"
server_key: "66611b42168c9b182ab703a20ebf4e66"
server_key_iv: "f7eea546a6ed391c1a0fdc999b9e703a"
server_key_version: "encryptionkey@0"
I20260620 18:43:20.578651 12044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980676253068-24012-0/minicluster-data/master-0/wal/instance:
uuid: "e481c921702a47eb8a2989431c1ec9d6"
format_stamp: "Formatted at 2026-06-20 18:43:20 on dist-test-slave-d645"
server_key: "66611b42168c9b182ab703a20ebf4e66"
server_key_iv: "f7eea546a6ed391c1a0fdc999b9e703a"
server_key_version: "encryptionkey@0"
I20260620 18:43:20.583638 12044 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260620 18:43:20.587054 12059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:20.588326 12044 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 18:43:20.588485 12044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980676253068-24012-0/minicluster-data/master-0/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980676253068-24012-0/minicluster-data/master-0/wal
uuid: "e481c921702a47eb8a2989431c1ec9d6"
format_stamp: "Formatted at 2026-06-20 18:43:20 on dist-test-slave-d645"
server_key: "66611b42168c9b182ab703a20ebf4e66"
server_key_iv: "f7eea546a6ed391c1a0fdc999b9e703a"
server_key_version: "encryptionkey@0"
I20260620 18:43:20.588654 12044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980676253068-24012-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980676253068-24012-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980676253068-24012-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:43:20.602907 12044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:43:20.607542 12044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:43:20.607800 12044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:43:20.623509 12044 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35489
I20260620 18:43:20.623538 12110 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35489 every 8 connection(s)
I20260620 18:43:20.624763 12044 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
I20260620 18:43:20.628238 12111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:43:20.634748 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 12044
I20260620 18:43:20.634958 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980676253068-24012-0/minicluster-data/master-0/wal/instance
I20260620 18:43:20.635524 24012 external_mini_cluster.cc:1428] Setting key 4c4b31683ca6b132009d29882495644c
I20260620 18:43:20.638996 12111 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e481c921702a47eb8a2989431c1ec9d6: Bootstrap starting.
I20260620 18:43:20.642063 12111 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e481c921702a47eb8a2989431c1ec9d6: Neither blocks nor log segments found. Creating new log.
I20260620 18:43:20.643194 12111 log.cc:824] T 00000000000000000000000000000000 P e481c921702a47eb8a2989431c1ec9d6: Log is configured to *not* fsync() on all Append() calls
I20260620 18:43:20.646394 12111 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e481c921702a47eb8a2989431c1ec9d6: No bootstrap required, opened a new log
I20260620 18:43:20.648705 12114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:20.637251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42821 (local address 127.0.0.1:35489)
0620 18:43:20.638094 (+ 843us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:20.638108 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:20.638156 (+ 48us) server_negotiation.cc:407] Connection header received
0620 18:43:20.640030 (+ 1874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:20.640051 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:20.640425 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:20.640723 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:20.641241 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:20.642524 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:20.644209 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:20.644984 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:20.645132 (+ 148us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:20.646437 (+ 1305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:20.646450 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:20.646470 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:43:20.647028 (+ 558us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:20.647103 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:43:20.647136 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:20.647256 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:20.647577 (+ 321us) server_negotiation.cc:299] Negotiation successful
0620 18:43:20.647746 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":1093,"threads_started":1}
I20260620 18:43:20.652206 24012 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:43:20.655179 12111 raft_consensus.cc:348] T 00000000000000000000000000000000 P e481c921702a47eb8a2989431c1ec9d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e481c921702a47eb8a2989431c1ec9d6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35489 } }
I20260620 18:43:20.655663 12111 raft_consensus.cc:374] T 00000000000000000000000000000000 P e481c921702a47eb8a2989431c1ec9d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:43:20.655787 12111 raft_consensus.cc:729] T 00000000000000000000000000000000 P e481c921702a47eb8a2989431c1ec9d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e481c921702a47eb8a2989431c1ec9d6, State: Initialized, Role: FOLLOWER
I20260620 18:43:20.656466 12111 consensus_queue.cc:260] T 00000000000000000000000000000000 P e481c921702a47eb8a2989431c1ec9d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e481c921702a47eb8a2989431c1ec9d6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35489 } }
I20260620 18:43:20.656754 12111 raft_consensus.cc:388] T 00000000000000000000000000000000 P e481c921702a47eb8a2989431c1ec9d6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:43:20.656898 12111 raft_consensus.cc:482] T 00000000000000000000000000000000 P e481c921702a47eb8a2989431c1ec9d6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:43:20.657069 12111 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e481c921702a47eb8a2989431c1ec9d6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:43:20.659922 12111 raft_consensus.cc:504] T 00000000000000000000000000000000 P e481c921702a47eb8a2989431c1ec9d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e481c921702a47eb8a2989431c1ec9d6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35489 } }
I20260620 18:43:20.660501 12111 leader_election.cc:302] T 00000000000000000000000000000000 P e481c921702a47eb8a2989431c1ec9d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e481c921702a47eb8a2989431c1ec9d6; no voters:
I20260620 18:43:20.661170 12111 leader_election.cc:288] T 00000000000000000000000000000000 P e481c921702a47eb8a2989431c1ec9d6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:43:20.661330 12121 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e481c921702a47eb8a2989431c1ec9d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:43:20.662081 12121 raft_consensus.cc:686] T 00000000000000000000000000000000 P e481c921702a47eb8a2989431c1ec9d6 [term 1 LEADER]: Becoming Leader. State: Replica: e481c921702a47eb8a2989431c1ec9d6, State: Running, Role: LEADER
I20260620 18:43:20.662850 12121 consensus_queue.cc:237] T 00000000000000000000000000000000 P e481c921702a47eb8a2989431c1ec9d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e481c921702a47eb8a2989431c1ec9d6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35489 } }
I20260620 18:43:20.663513 12111 sys_catalog.cc:564] T 00000000000000000000000000000000 P e481c921702a47eb8a2989431c1ec9d6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:43:20.665095 12114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:20.657532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43503 (local address 127.0.0.1:35489)
0620 18:43:20.657775 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:20.657784 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:20.658022 (+ 238us) server_negotiation.cc:407] Connection header received
0620 18:43:20.658177 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:20.658181 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:20.658256 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:20.658334 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:20.658991 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:20.659773 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:20.661641 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:20.662494 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:20.662644 (+ 150us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:20.664034 (+ 1390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:20.664044 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:20.664050 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:43:20.664073 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:20.664150 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:43:20.664176 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:20.664320 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:20.664560 (+ 240us) server_negotiation.cc:299] Negotiation successful
0620 18:43:20.664626 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":76}
I20260620 18:43:20.668224 12123 sys_catalog.cc:455] T 00000000000000000000000000000000 P e481c921702a47eb8a2989431c1ec9d6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e481c921702a47eb8a2989431c1ec9d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e481c921702a47eb8a2989431c1ec9d6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35489 } } }
I20260620 18:43:20.668665 12123 sys_catalog.cc:458] T 00000000000000000000000000000000 P e481c921702a47eb8a2989431c1ec9d6 [sys.catalog]: This master's current role is: LEADER
I20260620 18:43:20.669817 12122 sys_catalog.cc:455] T 00000000000000000000000000000000 P e481c921702a47eb8a2989431c1ec9d6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e481c921702a47eb8a2989431c1ec9d6. Latest consensus state: current_term: 1 leader_uuid: "e481c921702a47eb8a2989431c1ec9d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e481c921702a47eb8a2989431c1ec9d6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35489 } } }
I20260620 18:43:20.670071 12122 sys_catalog.cc:458] T 00000000000000000000000000000000 P e481c921702a47eb8a2989431c1ec9d6 [sys.catalog]: This master's current role is: LEADER
I20260620 18:43:20.670614 12135 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:43:20.675066 12135 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:43:20.682955 12135 catalog_manager.cc:1347] Generated new cluster ID: cc90e6bc78b4486997c3f75c5b77e5a4
I20260620 18:43:20.683077 12135 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:43:20.711346 12135 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:43:20.713286 12135 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:43:20.725179 12135 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e481c921702a47eb8a2989431c1ec9d6: Generated new TSK 0
I20260620 18:43:20.726213 12135 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:43:20.735499 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 12044
W20260620 18:43:20.736703 12106 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:42821
W20260620 18:43:20.737424 24012 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 12044: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:43:20.737589 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 12044
W20260620 18:43:20.751485 12029 connection.cc:535] client connection to 127.0.0.1:35489 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35489): Connection reset by peer (error 104)
2026-06-20T18:43:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1784 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980676253068-24012-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:43:20 dist-test-slave-d645 krb5kdc[12143](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:43:20 dist-test-slave-d645 krb5kdc[12143](info): set up 2 sockets
Jun 20 18:43:20 dist-test-slave-d645 krb5kdc[12143](info): commencing operation
krb5kdc: starting...
W20260620 18:43:21.767299 24012 mini_kdc.cc:121] Time spent starting KDC: real 0.997s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980676253068-24012-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980676253068-24012-0/krb5kdc/test-user.keytab.
Jun 20 18:43:21 dist-test-slave-d645 krb5kdc[12143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981001, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:43:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:43:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980676253068-24012-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980676253068-24012-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 18:43:22.002202 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980676253068-24012-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980676253068-24012-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980676253068-24012-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36999
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980676253068-24012-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36999
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:43:22.328760 12156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:43:22.329039 12156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:43:22.329173 12156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:43:22.336313 12156 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:43:22.336440 12156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:43:22.336516 12156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:43:22.336598 12156 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:43:22.336658 12156 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:43:22.346922 12156 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980676253068-24012-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980676253068-24012-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36999
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980676253068-24012-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36999
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980676253068-24012-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:43:22.348207 12156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:43:22.349434 12156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:43:22.360477 12164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:22.360517 12162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:22.361055 12161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:22.361357 12156 server_base.cc:1034] running on GCE node
I20260620 18:43:22.362069 12156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:43:22.363471 12156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:43:22.364687 12156 hybrid_clock.cc:648] HybridClock initialized: now 1781981002364635 us; error 45 us; skew 500 ppm
I20260620 18:43:22.365033 12156 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:43:22 dist-test-slave-d645 krb5kdc[12143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981002, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:43:22.371021 12156 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 18:43:22.372493 12156 webserver.cc:460] Webserver started at http://127.0.0.1:37451/ using document root <none> and password file <none>
I20260620 18:43:22.373127 12156 fs_manager.cc:359] Metadata directory not provided
I20260620 18:43:22.373243 12156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:43:22.373519 12156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:43:22.376387 12156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980676253068-24012-0/minicluster-data/master-0/data/instance:
uuid: "256baad767654cacaa9becc52db551a8"
format_stamp: "Formatted at 2026-06-20 18:43:22 on dist-test-slave-d645"
server_key: "761c0f69b53f73fbe182893e92c64478"
server_key_iv: "cd9638378a112e632bf167929c24a865"
server_key_version: "encryptionkey@0"
I20260620 18:43:22.377099 12156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980676253068-24012-0/minicluster-data/master-0/wal/instance:
uuid: "256baad767654cacaa9becc52db551a8"
format_stamp: "Formatted at 2026-06-20 18:43:22 on dist-test-slave-d645"
server_key: "761c0f69b53f73fbe182893e92c64478"
server_key_iv: "cd9638378a112e632bf167929c24a865"
server_key_version: "encryptionkey@0"
I20260620 18:43:22.381865 12156 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260620 18:43:22.385182 12171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:22.386451 12156 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:43:22.386605 12156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980676253068-24012-0/minicluster-data/master-0/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980676253068-24012-0/minicluster-data/master-0/wal
uuid: "256baad767654cacaa9becc52db551a8"
format_stamp: "Formatted at 2026-06-20 18:43:22 on dist-test-slave-d645"
server_key: "761c0f69b53f73fbe182893e92c64478"
server_key_iv: "cd9638378a112e632bf167929c24a865"
server_key_version: "encryptionkey@0"
I20260620 18:43:22.386772 12156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980676253068-24012-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980676253068-24012-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980676253068-24012-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:43:22.392125 12156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:43:22.396111 12156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:43:22.396363 12156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:43:22.413234 12156 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36999
I20260620 18:43:22.413272 12222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36999 every 8 connection(s)
I20260620 18:43:22.414532 12156 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
I20260620 18:43:22.416404 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 12156
I20260620 18:43:22.416567 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980676253068-24012-0/minicluster-data/master-0/wal/instance
I20260620 18:43:22.416991 24012 external_mini_cluster.cc:1428] Setting key 5c3625439f1559d1cba8a314b8ec6e52
I20260620 18:43:22.418403 12223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:43:22.423977 12226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:22.418585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37677 (local address 127.0.0.1:36999)
0620 18:43:22.419339 (+ 754us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:22.419350 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:22.419389 (+ 39us) server_negotiation.cc:407] Connection header received
0620 18:43:22.420885 (+ 1496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:22.420905 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:22.421225 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:22.421531 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:22.422531 (+ 1000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:22.422537 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:22.422548 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:43:22.422561 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:22.422630 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:43:22.422649 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:22.422806 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:22.422918 (+ 112us) server_negotiation.cc:299] Negotiation successful
0620 18:43:22.423125 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":343,"threads_started":1}
I20260620 18:43:22.427161 24012 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:43:22.430804 12223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 256baad767654cacaa9becc52db551a8: Bootstrap starting.
I20260620 18:43:22.434540 12223 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 256baad767654cacaa9becc52db551a8: Neither blocks nor log segments found. Creating new log.
I20260620 18:43:22.434698 12226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:22.432334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48241 (local address 127.0.0.1:36999)
0620 18:43:22.432514 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:22.432520 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:22.432733 (+ 213us) server_negotiation.cc:407] Connection header received
0620 18:43:22.432956 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:22.432961 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:22.433010 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:22.433102 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:22.433932 (+ 830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:22.433936 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:22.433940 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:43:22.433956 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:22.434001 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:43:22.434017 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:22.434113 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:22.434244 (+ 131us) server_negotiation.cc:299] Negotiation successful
0620 18:43:22.434361 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 18:43:22.435714 12223 log.cc:824] T 00000000000000000000000000000000 P 256baad767654cacaa9becc52db551a8: Log is configured to *not* fsync() on all Append() calls
I20260620 18:43:22.438123 12223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 256baad767654cacaa9becc52db551a8: No bootstrap required, opened a new log
I20260620 18:43:22.445245 12223 raft_consensus.cc:348] T 00000000000000000000000000000000 P 256baad767654cacaa9becc52db551a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "256baad767654cacaa9becc52db551a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36999 } }
I20260620 18:43:22.445559 12223 raft_consensus.cc:374] T 00000000000000000000000000000000 P 256baad767654cacaa9becc52db551a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:43:22.445691 12223 raft_consensus.cc:729] T 00000000000000000000000000000000 P 256baad767654cacaa9becc52db551a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 256baad767654cacaa9becc52db551a8, State: Initialized, Role: FOLLOWER
I20260620 18:43:22.446254 12223 consensus_queue.cc:260] T 00000000000000000000000000000000 P 256baad767654cacaa9becc52db551a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "256baad767654cacaa9becc52db551a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36999 } }
I20260620 18:43:22.446465 12223 raft_consensus.cc:388] T 00000000000000000000000000000000 P 256baad767654cacaa9becc52db551a8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:43:22.446600 12223 raft_consensus.cc:482] T 00000000000000000000000000000000 P 256baad767654cacaa9becc52db551a8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:43:22.446748 12223 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 256baad767654cacaa9becc52db551a8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:43:22.448832 12223 raft_consensus.cc:504] T 00000000000000000000000000000000 P 256baad767654cacaa9becc52db551a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "256baad767654cacaa9becc52db551a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36999 } }
I20260620 18:43:22.449291 12223 leader_election.cc:302] T 00000000000000000000000000000000 P 256baad767654cacaa9becc52db551a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 256baad767654cacaa9becc52db551a8; no voters:
I20260620 18:43:22.449848 12223 leader_election.cc:288] T 00000000000000000000000000000000 P 256baad767654cacaa9becc52db551a8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:43:22.450017 12233 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 256baad767654cacaa9becc52db551a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:43:22.450714 12233 raft_consensus.cc:686] T 00000000000000000000000000000000 P 256baad767654cacaa9becc52db551a8 [term 1 LEADER]: Becoming Leader. State: Replica: 256baad767654cacaa9becc52db551a8, State: Running, Role: LEADER
I20260620 18:43:22.451239 12233 consensus_queue.cc:237] T 00000000000000000000000000000000 P 256baad767654cacaa9becc52db551a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "256baad767654cacaa9becc52db551a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36999 } }
I20260620 18:43:22.451826 12223 sys_catalog.cc:564] T 00000000000000000000000000000000 P 256baad767654cacaa9becc52db551a8 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:43:22.453651 12235 sys_catalog.cc:455] T 00000000000000000000000000000000 P 256baad767654cacaa9becc52db551a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 256baad767654cacaa9becc52db551a8. Latest consensus state: current_term: 1 leader_uuid: "256baad767654cacaa9becc52db551a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "256baad767654cacaa9becc52db551a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36999 } } }
I20260620 18:43:22.454353 12235 sys_catalog.cc:458] T 00000000000000000000000000000000 P 256baad767654cacaa9becc52db551a8 [sys.catalog]: This master's current role is: LEADER
I20260620 18:43:22.454233 12234 sys_catalog.cc:455] T 00000000000000000000000000000000 P 256baad767654cacaa9becc52db551a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "256baad767654cacaa9becc52db551a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "256baad767654cacaa9becc52db551a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36999 } } }
I20260620 18:43:22.454699 12234 sys_catalog.cc:458] T 00000000000000000000000000000000 P 256baad767654cacaa9becc52db551a8 [sys.catalog]: This master's current role is: LEADER
I20260620 18:43:22.460606 12241 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:43:22.465480 12241 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:43:22.473923 12241 catalog_manager.cc:1347] Generated new cluster ID: f6c38c357ed04fcd88b1550fbefc613a
I20260620 18:43:22.474041 12241 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:43:22.486685 12241 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:43:22.487185 12241 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:43:22.493449 12241 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 256baad767654cacaa9becc52db551a8: Generated new TSK 0
I20260620 18:43:22.494190 12241 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:43:22.501711 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 12156
W20260620 18:43:22.502744 12218 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:37677
W20260620 18:43:22.503418 24012 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 12156: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:43:22.503572 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 12156
2026-06-20T18:43:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1766 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980676253068-24012-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:43:22 dist-test-slave-d645 krb5kdc[12254](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:43:22 dist-test-slave-d645 krb5kdc[12254](info): set up 2 sockets
Jun 20 18:43:22 dist-test-slave-d645 krb5kdc[12254](info): commencing operation
krb5kdc: starting...
W20260620 18:43:23.527365 24012 mini_kdc.cc:121] Time spent starting KDC: real 0.991s user 0.003s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980676253068-24012-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980676253068-24012-0/krb5kdc/test-user.keytab.
Jun 20 18:43:23 dist-test-slave-d645 krb5kdc[12254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981003, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:43:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:43:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980676253068-24012-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980676253068-24012-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 18:43:23.758785 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980676253068-24012-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980676253068-24012-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980676253068-24012-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38459
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980676253068-24012-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38459
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:43:24.076769 12267 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:43:24.077062 12267 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:43:24.077203 12267 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:43:24.084324 12267 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:43:24.084458 12267 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:43:24.084542 12267 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:43:24.084621 12267 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:43:24.084678 12267 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:43:24.094515 12267 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980676253068-24012-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980676253068-24012-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38459
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980676253068-24012-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38459
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980676253068-24012-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:43:24.095799 12267 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:43:24.097028 12267 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:43:24.108410 12272 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:24.108342 12275 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:24.108389 12273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:24.109318 12267 server_base.cc:1034] running on GCE node
I20260620 18:43:24.110021 12267 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:43:24.111429 12267 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:43:24.112637 12267 hybrid_clock.cc:648] HybridClock initialized: now 1781981004112599 us; error 29 us; skew 500 ppm
I20260620 18:43:24.112977 12267 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:43:24 dist-test-slave-d645 krb5kdc[12254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981004, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:43:24.115993 12267 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 18:43:24.117363 12267 webserver.cc:460] Webserver started at http://127.0.0.1:43777/ using document root <none> and password file <none>
I20260620 18:43:24.117986 12267 fs_manager.cc:359] Metadata directory not provided
I20260620 18:43:24.118098 12267 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:43:24.118368 12267 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:43:24.121201 12267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980676253068-24012-0/minicluster-data/master-0/data/instance:
uuid: "eeabb84ed6e747f8a14d6b9016916a4e"
format_stamp: "Formatted at 2026-06-20 18:43:24 on dist-test-slave-d645"
server_key: "0176a33b36ec7baf97b979cd593755cb"
server_key_iv: "9aa1f73895350e3e10af62414817076f"
server_key_version: "encryptionkey@0"
I20260620 18:43:24.121912 12267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980676253068-24012-0/minicluster-data/master-0/wal/instance:
uuid: "eeabb84ed6e747f8a14d6b9016916a4e"
format_stamp: "Formatted at 2026-06-20 18:43:24 on dist-test-slave-d645"
server_key: "0176a33b36ec7baf97b979cd593755cb"
server_key_iv: "9aa1f73895350e3e10af62414817076f"
server_key_version: "encryptionkey@0"
I20260620 18:43:24.126648 12267 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.002s
I20260620 18:43:24.129920 12282 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:24.131561 12267 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 18:43:24.131724 12267 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980676253068-24012-0/minicluster-data/master-0/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980676253068-24012-0/minicluster-data/master-0/wal
uuid: "eeabb84ed6e747f8a14d6b9016916a4e"
format_stamp: "Formatted at 2026-06-20 18:43:24 on dist-test-slave-d645"
server_key: "0176a33b36ec7baf97b979cd593755cb"
server_key_iv: "9aa1f73895350e3e10af62414817076f"
server_key_version: "encryptionkey@0"
I20260620 18:43:24.131906 12267 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980676253068-24012-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980676253068-24012-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980676253068-24012-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:43:24.165297 12267 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:43:24.169307 12267 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:43:24.169593 12267 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:43:24.187456 12267 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38459
I20260620 18:43:24.187492 12333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38459 every 8 connection(s)
I20260620 18:43:24.188727 12267 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
I20260620 18:43:24.191689 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 12267
I20260620 18:43:24.191874 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980676253068-24012-0/minicluster-data/master-0/wal/instance
I20260620 18:43:24.192305 24012 external_mini_cluster.cc:1428] Setting key 2b5c89111cc65185bd9353e7731d7fe1
I20260620 18:43:24.192310 12334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:43:24.206008 12334 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eeabb84ed6e747f8a14d6b9016916a4e: Bootstrap starting.
Jun 20 18:43:24 dist-test-slave-d645 krb5kdc[12254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981003, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 18:43:24.209616 12334 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eeabb84ed6e747f8a14d6b9016916a4e: Neither blocks nor log segments found. Creating new log.
I20260620 18:43:24.210891 12334 log.cc:824] T 00000000000000000000000000000000 P eeabb84ed6e747f8a14d6b9016916a4e: Log is configured to *not* fsync() on all Append() calls
I20260620 18:43:24.214360 12334 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eeabb84ed6e747f8a14d6b9016916a4e: No bootstrap required, opened a new log
I20260620 18:43:24.216531 12337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:24.193957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39807 (local address 127.0.0.1:38459)
0620 18:43:24.194765 (+ 808us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:24.194779 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:24.194824 (+ 45us) server_negotiation.cc:407] Connection header received
0620 18:43:24.196945 (+ 2121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:24.196972 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:24.197484 (+ 512us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:24.197895 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:24.199260 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:24.201149 (+ 1889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:24.202397 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:24.203458 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:24.203638 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:24.207963 (+ 4325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:24.207974 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:24.207993 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:24.208037 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:24.211951 (+ 3914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:24.212721 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:24.212731 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:24.212738 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:24.212966 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:24.213333 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:24.213337 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:24.213340 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:24.214102 (+ 762us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:24.214414 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:24.214796 (+ 382us) server_negotiation.cc:299] Negotiation successful
0620 18:43:24.215264 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":329,"threads_started":1}
I20260620 18:43:24.220449 24012 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:43:24.223258 12334 raft_consensus.cc:348] T 00000000000000000000000000000000 P eeabb84ed6e747f8a14d6b9016916a4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eeabb84ed6e747f8a14d6b9016916a4e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38459 } }
I20260620 18:43:24.223632 12334 raft_consensus.cc:374] T 00000000000000000000000000000000 P eeabb84ed6e747f8a14d6b9016916a4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:43:24.223726 12334 raft_consensus.cc:729] T 00000000000000000000000000000000 P eeabb84ed6e747f8a14d6b9016916a4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeabb84ed6e747f8a14d6b9016916a4e, State: Initialized, Role: FOLLOWER
I20260620 18:43:24.224279 12334 consensus_queue.cc:260] T 00000000000000000000000000000000 P eeabb84ed6e747f8a14d6b9016916a4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eeabb84ed6e747f8a14d6b9016916a4e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38459 } }
I20260620 18:43:24.224519 12334 raft_consensus.cc:388] T 00000000000000000000000000000000 P eeabb84ed6e747f8a14d6b9016916a4e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:43:24.224665 12334 raft_consensus.cc:482] T 00000000000000000000000000000000 P eeabb84ed6e747f8a14d6b9016916a4e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:43:24.224923 12334 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eeabb84ed6e747f8a14d6b9016916a4e [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:43:24.227880 12334 raft_consensus.cc:504] T 00000000000000000000000000000000 P eeabb84ed6e747f8a14d6b9016916a4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eeabb84ed6e747f8a14d6b9016916a4e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38459 } }
I20260620 18:43:24.228423 12334 leader_election.cc:302] T 00000000000000000000000000000000 P eeabb84ed6e747f8a14d6b9016916a4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eeabb84ed6e747f8a14d6b9016916a4e; no voters:
I20260620 18:43:24.229050 12334 leader_election.cc:288] T 00000000000000000000000000000000 P eeabb84ed6e747f8a14d6b9016916a4e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:43:24.229146 12344 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eeabb84ed6e747f8a14d6b9016916a4e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:43:24.231216 12334 sys_catalog.cc:564] T 00000000000000000000000000000000 P eeabb84ed6e747f8a14d6b9016916a4e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:43:24.236310 12344 raft_consensus.cc:686] T 00000000000000000000000000000000 P eeabb84ed6e747f8a14d6b9016916a4e [term 1 LEADER]: Becoming Leader. State: Replica: eeabb84ed6e747f8a14d6b9016916a4e, State: Running, Role: LEADER
I20260620 18:43:24.236991 12344 consensus_queue.cc:237] T 00000000000000000000000000000000 P eeabb84ed6e747f8a14d6b9016916a4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eeabb84ed6e747f8a14d6b9016916a4e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38459 } }
I20260620 18:43:24.239702 12345 sys_catalog.cc:455] T 00000000000000000000000000000000 P eeabb84ed6e747f8a14d6b9016916a4e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eeabb84ed6e747f8a14d6b9016916a4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eeabb84ed6e747f8a14d6b9016916a4e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38459 } } }
I20260620 18:43:24.240099 12345 sys_catalog.cc:458] T 00000000000000000000000000000000 P eeabb84ed6e747f8a14d6b9016916a4e [sys.catalog]: This master's current role is: LEADER
I20260620 18:43:24.241593 12337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:24.224678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45167 (local address 127.0.0.1:38459)
0620 18:43:24.224879 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:24.224886 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:24.225203 (+ 317us) server_negotiation.cc:407] Connection header received
0620 18:43:24.225370 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:24.225373 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:24.225435 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:24.225510 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:24.227315 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:24.228052 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:24.229848 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:24.230509 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:24.230611 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:24.233813 (+ 3202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:24.233821 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:24.233827 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:24.233869 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:24.237591 (+ 3722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:24.238348 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:24.238354 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:24.238357 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:24.238529 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:24.239028 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:24.239033 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:24.239040 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:24.239416 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:24.239674 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:24.240803 (+ 1129us) server_negotiation.cc:299] Negotiation successful
0620 18:43:24.241052 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 18:43:24.243060 12346 sys_catalog.cc:455] T 00000000000000000000000000000000 P eeabb84ed6e747f8a14d6b9016916a4e [sys.catalog]: SysCatalogTable state changed. Reason: New leader eeabb84ed6e747f8a14d6b9016916a4e. Latest consensus state: current_term: 1 leader_uuid: "eeabb84ed6e747f8a14d6b9016916a4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eeabb84ed6e747f8a14d6b9016916a4e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38459 } } }
I20260620 18:43:24.243432 12346 sys_catalog.cc:458] T 00000000000000000000000000000000 P eeabb84ed6e747f8a14d6b9016916a4e [sys.catalog]: This master's current role is: LEADER
I20260620 18:43:24.243939 12354 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:43:24.248819 12354 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:43:24.258899 12354 catalog_manager.cc:1347] Generated new cluster ID: 26eab9359e93487f8d786b14d23a1aa7
I20260620 18:43:24.259123 12354 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:43:24.271394 12354 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:43:24.273207 12354 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:43:24.282407 12354 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eeabb84ed6e747f8a14d6b9016916a4e: Generated new TSK 0
I20260620 18:43:24.283506 12354 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:43:24.298688 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 12267
I20260620 18:43:24.299705 12329 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:43:24.388154 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 12267
2026-06-20T18:43:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1884 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980676253068-24012-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:43:24 dist-test-slave-d645 krb5kdc[12367](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:43:24 dist-test-slave-d645 krb5kdc[12367](info): set up 2 sockets
Jun 20 18:43:24 dist-test-slave-d645 krb5kdc[12367](info): commencing operation
krb5kdc: starting...
W20260620 18:43:25.407282 24012 mini_kdc.cc:121] Time spent starting KDC: real 0.987s user 0.013s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980676253068-24012-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980676253068-24012-0/krb5kdc/test-user.keytab.
Jun 20 18:43:25 dist-test-slave-d645 krb5kdc[12367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981005, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:43:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:43:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980676253068-24012-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980676253068-24012-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 18:43:25.646212 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980676253068-24012-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980676253068-24012-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980676253068-24012-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44353
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980676253068-24012-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44353
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:43:25.982044 12380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:43:25.982363 12380 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:43:25.982503 12380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:43:25.989725 12380 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:43:25.989847 12380 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:43:25.989925 12380 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:43:25.990002 12380 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:43:25.990063 12380 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:43:25.999881 12380 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980676253068-24012-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980676253068-24012-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44353
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980676253068-24012-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44353
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980676253068-24012-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:43:26.001181 12380 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:43:26.002415 12380 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:43:26.014452 12385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:26.014325 12388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:26.014420 12386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:26.016460 12380 server_base.cc:1034] running on GCE node
I20260620 18:43:26.017380 12380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:43:26.019302 12380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:43:26.020563 12380 hybrid_clock.cc:648] HybridClock initialized: now 1781981006020502 us; error 51 us; skew 500 ppm
I20260620 18:43:26.020895 12380 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:43:26 dist-test-slave-d645 krb5kdc[12367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981006, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:43:26.024307 12380 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 18:43:26.025846 12380 webserver.cc:460] Webserver started at http://127.0.0.1:35765/ using document root <none> and password file <none>
I20260620 18:43:26.026623 12380 fs_manager.cc:359] Metadata directory not provided
I20260620 18:43:26.026757 12380 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:43:26.027094 12380 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:43:26.030292 12380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980676253068-24012-0/minicluster-data/master-0/data/instance:
uuid: "3e5169ea87ed4400aa6590326473c4be"
format_stamp: "Formatted at 2026-06-20 18:43:26 on dist-test-slave-d645"
server_key: "779a146961ae9743537ce0eeefb3d944"
server_key_iv: "eceec2e24e347ed65dce89622bcbc6de"
server_key_version: "encryptionkey@0"
I20260620 18:43:26.031134 12380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980676253068-24012-0/minicluster-data/master-0/wal/instance:
uuid: "3e5169ea87ed4400aa6590326473c4be"
format_stamp: "Formatted at 2026-06-20 18:43:26 on dist-test-slave-d645"
server_key: "779a146961ae9743537ce0eeefb3d944"
server_key_iv: "eceec2e24e347ed65dce89622bcbc6de"
server_key_version: "encryptionkey@0"
I20260620 18:43:26.036456 12380 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260620 18:43:26.040162 12395 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:26.041455 12380 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 18:43:26.041601 12380 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980676253068-24012-0/minicluster-data/master-0/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980676253068-24012-0/minicluster-data/master-0/wal
uuid: "3e5169ea87ed4400aa6590326473c4be"
format_stamp: "Formatted at 2026-06-20 18:43:26 on dist-test-slave-d645"
server_key: "779a146961ae9743537ce0eeefb3d944"
server_key_iv: "eceec2e24e347ed65dce89622bcbc6de"
server_key_version: "encryptionkey@0"
I20260620 18:43:26.041771 12380 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980676253068-24012-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980676253068-24012-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980676253068-24012-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:43:26.052114 12380 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:43:26.056169 12380 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:43:26.056394 12380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:43:26.073915 12380 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44353
I20260620 18:43:26.073966 12446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44353 every 8 connection(s)
I20260620 18:43:26.075278 12380 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
I20260620 18:43:26.078886 12447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:43:26.079775 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 12380
I20260620 18:43:26.079946 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980676253068-24012-0/minicluster-data/master-0/wal/instance
I20260620 18:43:26.080356 24012 external_mini_cluster.cc:1428] Setting key 5db03e434b84bd697956cac4c599f36e
I20260620 18:43:26.091754 12447 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e5169ea87ed4400aa6590326473c4be: Bootstrap starting.
I20260620 18:43:26.093196 12450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:26.081964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52719 (local address 127.0.0.1:44353)
0620 18:43:26.082666 (+ 702us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:26.082677 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:26.082719 (+ 42us) server_negotiation.cc:407] Connection header received
0620 18:43:26.084526 (+ 1807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:26.084551 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:26.084934 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:26.085260 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:26.085855 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:26.087185 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:26.088968 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:26.089684 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:26.089805 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:26.091022 (+ 1217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:26.091031 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:26.091044 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:43:26.091558 (+ 514us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:26.091630 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:43:26.091659 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:26.091743 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:26.092011 (+ 268us) server_negotiation.cc:299] Negotiation successful
0620 18:43:26.092265 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":313,"threads_started":1}
I20260620 18:43:26.094924 12447 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e5169ea87ed4400aa6590326473c4be: Neither blocks nor log segments found. Creating new log.
I20260620 18:43:26.096077 12447 log.cc:824] T 00000000000000000000000000000000 P 3e5169ea87ed4400aa6590326473c4be: Log is configured to *not* fsync() on all Append() calls
I20260620 18:43:26.096829 24012 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:43:26.098766 12447 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e5169ea87ed4400aa6590326473c4be: No bootstrap required, opened a new log
I20260620 18:43:26.106462 12450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:26.100985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51799 (local address 127.0.0.1:44353)
0620 18:43:26.101129 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:26.101134 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:26.101515 (+ 381us) server_negotiation.cc:407] Connection header received
0620 18:43:26.101686 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:26.101690 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:26.101751 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:26.101835 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:26.102343 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:26.103069 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:26.104303 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:26.104746 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:26.104848 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:26.105799 (+ 951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:26.105804 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:26.105807 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:43:26.105826 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:26.105859 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:43:26.105875 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:26.105958 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:26.106091 (+ 133us) server_negotiation.cc:299] Negotiation successful
0620 18:43:26.106176 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 18:43:26.106969 12447 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e5169ea87ed4400aa6590326473c4be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e5169ea87ed4400aa6590326473c4be" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44353 } }
I20260620 18:43:26.107362 12447 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3e5169ea87ed4400aa6590326473c4be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:43:26.107460 12447 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e5169ea87ed4400aa6590326473c4be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e5169ea87ed4400aa6590326473c4be, State: Initialized, Role: FOLLOWER
I20260620 18:43:26.108016 12447 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e5169ea87ed4400aa6590326473c4be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e5169ea87ed4400aa6590326473c4be" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44353 } }
I20260620 18:43:26.108229 12447 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3e5169ea87ed4400aa6590326473c4be [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:43:26.108369 12447 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e5169ea87ed4400aa6590326473c4be [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:43:26.108510 12447 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e5169ea87ed4400aa6590326473c4be [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:43:26.110579 12447 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e5169ea87ed4400aa6590326473c4be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e5169ea87ed4400aa6590326473c4be" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44353 } }
I20260620 18:43:26.111096 12447 leader_election.cc:302] T 00000000000000000000000000000000 P 3e5169ea87ed4400aa6590326473c4be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e5169ea87ed4400aa6590326473c4be; no voters:
I20260620 18:43:26.111642 12447 leader_election.cc:288] T 00000000000000000000000000000000 P 3e5169ea87ed4400aa6590326473c4be [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:43:26.111749 12457 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e5169ea87ed4400aa6590326473c4be [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:43:26.112349 12457 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e5169ea87ed4400aa6590326473c4be [term 1 LEADER]: Becoming Leader. State: Replica: 3e5169ea87ed4400aa6590326473c4be, State: Running, Role: LEADER
I20260620 18:43:26.112908 12457 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e5169ea87ed4400aa6590326473c4be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e5169ea87ed4400aa6590326473c4be" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44353 } }
I20260620 18:43:26.113585 12447 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e5169ea87ed4400aa6590326473c4be [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:43:26.115607 12459 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e5169ea87ed4400aa6590326473c4be [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e5169ea87ed4400aa6590326473c4be. Latest consensus state: current_term: 1 leader_uuid: "3e5169ea87ed4400aa6590326473c4be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e5169ea87ed4400aa6590326473c4be" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44353 } } }
I20260620 18:43:26.115983 12459 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e5169ea87ed4400aa6590326473c4be [sys.catalog]: This master's current role is: LEADER
I20260620 18:43:26.116663 12458 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e5169ea87ed4400aa6590326473c4be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3e5169ea87ed4400aa6590326473c4be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e5169ea87ed4400aa6590326473c4be" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44353 } } }
I20260620 18:43:26.116871 12465 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:43:26.117007 12458 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e5169ea87ed4400aa6590326473c4be [sys.catalog]: This master's current role is: LEADER
I20260620 18:43:26.122579 12465 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:43:26.131724 12465 catalog_manager.cc:1347] Generated new cluster ID: c723a1565c30428dab56cc28f98902d5
I20260620 18:43:26.131860 12465 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:43:26.148499 12465 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:43:26.149780 12465 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:43:26.157464 12465 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3e5169ea87ed4400aa6590326473c4be: Generated new TSK 0
I20260620 18:43:26.158257 12465 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:43:26.168609 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 12380
W20260620 18:43:26.169663 12442 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:52719
W20260620 18:43:26.170393 24012 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 12380: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:43:26.170516 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 12380
2026-06-20T18:43:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1783 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980676253068-24012-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:43:26 dist-test-slave-d645 krb5kdc[12478](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:43:26 dist-test-slave-d645 krb5kdc[12478](info): set up 2 sockets
Jun 20 18:43:26 dist-test-slave-d645 krb5kdc[12478](info): commencing operation
krb5kdc: starting...
W20260620 18:43:27.199793 24012 mini_kdc.cc:121] Time spent starting KDC: real 0.995s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980676253068-24012-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980676253068-24012-0/krb5kdc/test-user.keytab.
Jun 20 18:43:27 dist-test-slave-d645 krb5kdc[12478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981007, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:43:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:43:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980676253068-24012-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980676253068-24012-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 18:43:27.445837 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980676253068-24012-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980676253068-24012-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980676253068-24012-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35473
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980676253068-24012-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35473
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:43:27.770536 12491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:43:27.770850 12491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:43:27.770996 12491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:43:27.778664 12491 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:43:27.778786 12491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:43:27.778831 12491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:43:27.778900 12491 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:43:27.778986 12491 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:43:27.791450 12491 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980676253068-24012-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980676253068-24012-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35473
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980676253068-24012-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35473
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980676253068-24012-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:43:27.792694 12491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:43:27.793895 12491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:43:27.804980 12496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:27.805270 12499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:27.805307 12497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:27.806708 12491 server_base.cc:1034] running on GCE node
I20260620 18:43:27.807536 12491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:43:27.808908 12491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:43:27.810169 12491 hybrid_clock.cc:648] HybridClock initialized: now 1781981007810126 us; error 29 us; skew 500 ppm
I20260620 18:43:27.810555 12491 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:43:27 dist-test-slave-d645 krb5kdc[12478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981007, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:43:27.818349 12491 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 18:43:27.819938 12491 webserver.cc:460] Webserver started at http://127.0.0.1:34115/ using document root <none> and password file <none>
I20260620 18:43:27.820633 12491 fs_manager.cc:359] Metadata directory not provided
I20260620 18:43:27.820749 12491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:43:27.821033 12491 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:43:27.824057 12491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980676253068-24012-0/minicluster-data/master-0/data/instance:
uuid: "ca0797af231c405baa1c136e3cc010f7"
format_stamp: "Formatted at 2026-06-20 18:43:27 on dist-test-slave-d645"
server_key: "297b7a27d90dd978f0e7d917bdc5804e"
server_key_iv: "9f5bdd8a4b88cc66682e0646a7521b43"
server_key_version: "encryptionkey@0"
I20260620 18:43:27.824815 12491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980676253068-24012-0/minicluster-data/master-0/wal/instance:
uuid: "ca0797af231c405baa1c136e3cc010f7"
format_stamp: "Formatted at 2026-06-20 18:43:27 on dist-test-slave-d645"
server_key: "297b7a27d90dd978f0e7d917bdc5804e"
server_key_iv: "9f5bdd8a4b88cc66682e0646a7521b43"
server_key_version: "encryptionkey@0"
I20260620 18:43:27.829945 12491 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 18:43:27.833420 12506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:27.834766 12491 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:43:27.834981 12491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980676253068-24012-0/minicluster-data/master-0/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980676253068-24012-0/minicluster-data/master-0/wal
uuid: "ca0797af231c405baa1c136e3cc010f7"
format_stamp: "Formatted at 2026-06-20 18:43:27 on dist-test-slave-d645"
server_key: "297b7a27d90dd978f0e7d917bdc5804e"
server_key_iv: "9f5bdd8a4b88cc66682e0646a7521b43"
server_key_version: "encryptionkey@0"
I20260620 18:43:27.835175 12491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980676253068-24012-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980676253068-24012-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980676253068-24012-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:43:27.840979 12491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:43:27.845233 12491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:43:27.845503 12491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:43:27.863565 12491 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35473
I20260620 18:43:27.863777 12557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35473 every 8 connection(s)
I20260620 18:43:27.865041 12491 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
I20260620 18:43:27.868878 12558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:43:27.871258 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 12491
I20260620 18:43:27.871438 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980676253068-24012-0/minicluster-data/master-0/wal/instance
I20260620 18:43:27.871954 24012 external_mini_cluster.cc:1428] Setting key 0351500df327f352dacdf33d97efaa64
I20260620 18:43:27.880031 12561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:27.874283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34067 (local address 127.0.0.1:35473)
0620 18:43:27.875109 (+ 826us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:27.875120 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:27.875161 (+ 41us) server_negotiation.cc:407] Connection header received
0620 18:43:27.876795 (+ 1634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:27.876816 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:27.877143 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:27.877476 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:27.878536 (+ 1060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:27.878543 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:27.878558 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:43:27.878572 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:27.878633 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:43:27.878655 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:27.878826 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:27.878917 (+ 91us) server_negotiation.cc:299] Negotiation successful
0620 18:43:27.879124 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":361,"threads_started":1}
I20260620 18:43:27.880745 12558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca0797af231c405baa1c136e3cc010f7: Bootstrap starting.
I20260620 18:43:27.883769 24012 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:43:27.884006 12558 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca0797af231c405baa1c136e3cc010f7: Neither blocks nor log segments found. Creating new log.
I20260620 18:43:27.884955 12558 log.cc:824] T 00000000000000000000000000000000 P ca0797af231c405baa1c136e3cc010f7: Log is configured to *not* fsync() on all Append() calls
I20260620 18:43:27.889147 12558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca0797af231c405baa1c136e3cc010f7: No bootstrap required, opened a new log
I20260620 18:43:27.890625 12561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:27.888519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55417 (local address 127.0.0.1:35473)
0620 18:43:27.888659 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:27.888663 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:27.888808 (+ 145us) server_negotiation.cc:407] Connection header received
0620 18:43:27.889017 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:27.889021 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:27.889079 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:27.889168 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:27.890038 (+ 870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:27.890042 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:27.890046 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:43:27.890060 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:27.890100 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:43:27.890114 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:27.890200 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:27.890312 (+ 112us) server_negotiation.cc:299] Negotiation successful
0620 18:43:27.890359 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 18:43:27.897141 12558 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca0797af231c405baa1c136e3cc010f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca0797af231c405baa1c136e3cc010f7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35473 } }
I20260620 18:43:27.897511 12558 raft_consensus.cc:374] T 00000000000000000000000000000000 P ca0797af231c405baa1c136e3cc010f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:43:27.897583 12558 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca0797af231c405baa1c136e3cc010f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca0797af231c405baa1c136e3cc010f7, State: Initialized, Role: FOLLOWER
I20260620 18:43:27.898067 12558 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca0797af231c405baa1c136e3cc010f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca0797af231c405baa1c136e3cc010f7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35473 } }
I20260620 18:43:27.898272 12558 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca0797af231c405baa1c136e3cc010f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:43:27.898370 12558 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca0797af231c405baa1c136e3cc010f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:43:27.898494 12558 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca0797af231c405baa1c136e3cc010f7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:43:27.900840 12558 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca0797af231c405baa1c136e3cc010f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca0797af231c405baa1c136e3cc010f7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35473 } }
I20260620 18:43:27.901611 12558 leader_election.cc:302] T 00000000000000000000000000000000 P ca0797af231c405baa1c136e3cc010f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca0797af231c405baa1c136e3cc010f7; no voters:
I20260620 18:43:27.902148 12558 leader_election.cc:288] T 00000000000000000000000000000000 P ca0797af231c405baa1c136e3cc010f7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:43:27.902300 12568 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca0797af231c405baa1c136e3cc010f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:43:27.902964 12568 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca0797af231c405baa1c136e3cc010f7 [term 1 LEADER]: Becoming Leader. State: Replica: ca0797af231c405baa1c136e3cc010f7, State: Running, Role: LEADER
I20260620 18:43:27.903465 12568 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca0797af231c405baa1c136e3cc010f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca0797af231c405baa1c136e3cc010f7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35473 } }
I20260620 18:43:27.904194 12558 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca0797af231c405baa1c136e3cc010f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:43:27.906157 12570 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca0797af231c405baa1c136e3cc010f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca0797af231c405baa1c136e3cc010f7. Latest consensus state: current_term: 1 leader_uuid: "ca0797af231c405baa1c136e3cc010f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca0797af231c405baa1c136e3cc010f7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35473 } } }
I20260620 18:43:27.906594 12570 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca0797af231c405baa1c136e3cc010f7 [sys.catalog]: This master's current role is: LEADER
I20260620 18:43:27.909597 12575 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:43:27.909538 12569 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca0797af231c405baa1c136e3cc010f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ca0797af231c405baa1c136e3cc010f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca0797af231c405baa1c136e3cc010f7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35473 } } }
I20260620 18:43:27.909874 12569 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca0797af231c405baa1c136e3cc010f7 [sys.catalog]: This master's current role is: LEADER
I20260620 18:43:27.914870 12575 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:43:27.925186 12575 catalog_manager.cc:1347] Generated new cluster ID: 54dc99ed8e93424088560e5ce47d29ec
I20260620 18:43:27.925321 12575 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:43:27.938225 12575 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:43:27.938663 12575 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:43:27.950389 12575 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ca0797af231c405baa1c136e3cc010f7: Generated new TSK 0
I20260620 18:43:27.951560 12575 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:43:27.963243 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 12491
W20260620 18:43:27.964335 12553 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:34067
W20260620 18:43:27.965029 24012 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 12491: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:43:27.965178 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 12491
2026-06-20T18:43:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1794 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980676253068-24012-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:43:28 dist-test-slave-d645 krb5kdc[12589](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:43:28 dist-test-slave-d645 krb5kdc[12589](info): set up 2 sockets
Jun 20 18:43:28 dist-test-slave-d645 krb5kdc[12589](info): commencing operation
krb5kdc: starting...
W20260620 18:43:28.987409 24012 mini_kdc.cc:121] Time spent starting KDC: real 0.988s user 0.005s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980676253068-24012-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980676253068-24012-0/krb5kdc/test-user.keytab.
Jun 20 18:43:29 dist-test-slave-d645 krb5kdc[12589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981009, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:43:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:43:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980676253068-24012-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980676253068-24012-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 18:43:29.230854 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980676253068-24012-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980676253068-24012-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980676253068-24012-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42277
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980676253068-24012-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42277
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:43:29.550606 12602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:43:29.550876 12602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:43:29.551021 12602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:43:29.558427 12602 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:43:29.558539 12602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:43:29.558605 12602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:43:29.558687 12602 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:43:29.558748 12602 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:43:29.568418 12602 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980676253068-24012-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980676253068-24012-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42277
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980676253068-24012-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42277
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980676253068-24012-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:43:29.569914 12602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:43:29.571319 12602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:43:29.582424 12610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:29.582383 12608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:29.583006 12607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:29.583423 12602 server_base.cc:1034] running on GCE node
I20260620 18:43:29.584160 12602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:43:29.585533 12602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:43:29.586720 12602 hybrid_clock.cc:648] HybridClock initialized: now 1781981009586683 us; error 29 us; skew 500 ppm
I20260620 18:43:29.587081 12602 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:43:29 dist-test-slave-d645 krb5kdc[12589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981009, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:43:29.589929 12602 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 18:43:29.591332 12602 webserver.cc:460] Webserver started at http://127.0.0.1:41655/ using document root <none> and password file <none>
I20260620 18:43:29.591930 12602 fs_manager.cc:359] Metadata directory not provided
I20260620 18:43:29.592039 12602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:43:29.592290 12602 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:43:29.595083 12602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980676253068-24012-0/minicluster-data/master-0/data/instance:
uuid: "faaa4debfd134572aedc67bd4e1c0f3b"
format_stamp: "Formatted at 2026-06-20 18:43:29 on dist-test-slave-d645"
server_key: "19f5968333dc5a57c913e6f4edf3ca25"
server_key_iv: "4b1d5e6afad9765eed7a49e505f27319"
server_key_version: "encryptionkey@0"
I20260620 18:43:29.595781 12602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980676253068-24012-0/minicluster-data/master-0/wal/instance:
uuid: "faaa4debfd134572aedc67bd4e1c0f3b"
format_stamp: "Formatted at 2026-06-20 18:43:29 on dist-test-slave-d645"
server_key: "19f5968333dc5a57c913e6f4edf3ca25"
server_key_iv: "4b1d5e6afad9765eed7a49e505f27319"
server_key_version: "encryptionkey@0"
I20260620 18:43:29.601353 12602 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:43:29.604895 12617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:29.606379 12602 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:43:29.606556 12602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980676253068-24012-0/minicluster-data/master-0/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980676253068-24012-0/minicluster-data/master-0/wal
uuid: "faaa4debfd134572aedc67bd4e1c0f3b"
format_stamp: "Formatted at 2026-06-20 18:43:29 on dist-test-slave-d645"
server_key: "19f5968333dc5a57c913e6f4edf3ca25"
server_key_iv: "4b1d5e6afad9765eed7a49e505f27319"
server_key_version: "encryptionkey@0"
I20260620 18:43:29.606865 12602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980676253068-24012-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980676253068-24012-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980676253068-24012-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:43:29.621804 12602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:43:29.625859 12602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:43:29.626127 12602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:43:29.644008 12602 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42277
I20260620 18:43:29.644027 12668 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42277 every 8 connection(s)
I20260620 18:43:29.645371 12602 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
I20260620 18:43:29.648937 12669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:43:29.654223 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 12602
I20260620 18:43:29.654399 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980676253068-24012-0/minicluster-data/master-0/wal/instance
I20260620 18:43:29.654963 24012 external_mini_cluster.cc:1428] Setting key 33dfbca919f6707de339ccdec7d9e00f
I20260620 18:43:29.660069 12669 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P faaa4debfd134572aedc67bd4e1c0f3b: Bootstrap starting.
I20260620 18:43:29.663092 12669 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P faaa4debfd134572aedc67bd4e1c0f3b: Neither blocks nor log segments found. Creating new log.
I20260620 18:43:29.664065 12669 log.cc:824] T 00000000000000000000000000000000 P faaa4debfd134572aedc67bd4e1c0f3b: Log is configured to *not* fsync() on all Append() calls
I20260620 18:43:29.666922 12669 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P faaa4debfd134572aedc67bd4e1c0f3b: No bootstrap required, opened a new log
Jun 20 18:43:29 dist-test-slave-d645 krb5kdc[12589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981009, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 18:43:29.675164 12669 raft_consensus.cc:348] T 00000000000000000000000000000000 P faaa4debfd134572aedc67bd4e1c0f3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faaa4debfd134572aedc67bd4e1c0f3b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42277 } }
I20260620 18:43:29.675540 12669 raft_consensus.cc:374] T 00000000000000000000000000000000 P faaa4debfd134572aedc67bd4e1c0f3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:43:29.675647 12669 raft_consensus.cc:729] T 00000000000000000000000000000000 P faaa4debfd134572aedc67bd4e1c0f3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: faaa4debfd134572aedc67bd4e1c0f3b, State: Initialized, Role: FOLLOWER
I20260620 18:43:29.676240 12669 consensus_queue.cc:260] T 00000000000000000000000000000000 P faaa4debfd134572aedc67bd4e1c0f3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faaa4debfd134572aedc67bd4e1c0f3b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42277 } }
I20260620 18:43:29.676486 12669 raft_consensus.cc:388] T 00000000000000000000000000000000 P faaa4debfd134572aedc67bd4e1c0f3b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:43:29.676630 12669 raft_consensus.cc:482] T 00000000000000000000000000000000 P faaa4debfd134572aedc67bd4e1c0f3b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:43:29.676781 12669 raft_consensus.cc:3037] T 00000000000000000000000000000000 P faaa4debfd134572aedc67bd4e1c0f3b [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:43:29.677238 12672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:29.656808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.21:54149 (local address 127.0.0.1:42277)
0620 18:43:29.657614 (+ 806us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:29.657627 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:29.657673 (+ 46us) server_negotiation.cc:407] Connection header received
0620 18:43:29.659437 (+ 1764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:29.659459 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:29.659866 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:29.660200 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:29.661871 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:29.663193 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:29.664475 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:29.665184 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:29.665348 (+ 164us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:29.669312 (+ 3964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:29.669320 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:29.669335 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:29.669382 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:29.672961 (+ 3579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:29.673800 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:29.673806 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:29.673812 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:29.673987 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:29.674436 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:29.674439 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:29.674441 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:29.674984 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:29.675284 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:29.675761 (+ 477us) server_negotiation.cc:299] Negotiation successful
0620 18:43:29.676116 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":334,"threads_started":1}
I20260620 18:43:29.679234 12669 raft_consensus.cc:504] T 00000000000000000000000000000000 P faaa4debfd134572aedc67bd4e1c0f3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faaa4debfd134572aedc67bd4e1c0f3b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42277 } }
I20260620 18:43:29.679738 12669 leader_election.cc:302] T 00000000000000000000000000000000 P faaa4debfd134572aedc67bd4e1c0f3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: faaa4debfd134572aedc67bd4e1c0f3b; no voters:
I20260620 18:43:29.680392 12669 leader_election.cc:288] T 00000000000000000000000000000000 P faaa4debfd134572aedc67bd4e1c0f3b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:43:29.680500 12674 raft_consensus.cc:2781] T 00000000000000000000000000000000 P faaa4debfd134572aedc67bd4e1c0f3b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:43:29.681316 24012 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:43:29.681468 12674 raft_consensus.cc:686] T 00000000000000000000000000000000 P faaa4debfd134572aedc67bd4e1c0f3b [term 1 LEADER]: Becoming Leader. State: Replica: faaa4debfd134572aedc67bd4e1c0f3b, State: Running, Role: LEADER
I20260620 18:43:29.681988 12674 consensus_queue.cc:237] T 00000000000000000000000000000000 P faaa4debfd134572aedc67bd4e1c0f3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faaa4debfd134572aedc67bd4e1c0f3b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42277 } }
I20260620 18:43:29.682542 12669 sys_catalog.cc:564] T 00000000000000000000000000000000 P faaa4debfd134572aedc67bd4e1c0f3b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:43:29.684670 12675 sys_catalog.cc:455] T 00000000000000000000000000000000 P faaa4debfd134572aedc67bd4e1c0f3b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "faaa4debfd134572aedc67bd4e1c0f3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faaa4debfd134572aedc67bd4e1c0f3b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42277 } } }
I20260620 18:43:29.685043 12675 sys_catalog.cc:458] T 00000000000000000000000000000000 P faaa4debfd134572aedc67bd4e1c0f3b [sys.catalog]: This master's current role is: LEADER
I20260620 18:43:29.689653 12676 sys_catalog.cc:455] T 00000000000000000000000000000000 P faaa4debfd134572aedc67bd4e1c0f3b [sys.catalog]: SysCatalogTable state changed. Reason: New leader faaa4debfd134572aedc67bd4e1c0f3b. Latest consensus state: current_term: 1 leader_uuid: "faaa4debfd134572aedc67bd4e1c0f3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faaa4debfd134572aedc67bd4e1c0f3b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42277 } } }
I20260620 18:43:29.689940 12676 sys_catalog.cc:458] T 00000000000000000000000000000000 P faaa4debfd134572aedc67bd4e1c0f3b [sys.catalog]: This master's current role is: LEADER
I20260620 18:43:29.692237 12680 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:43:29.697041 12680 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:43:29.710497 12672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:29.690643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.21:56431 (local address 127.0.0.1:42277)
0620 18:43:29.690816 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:29.690823 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:29.692252 (+ 1429us) server_negotiation.cc:407] Connection header received
0620 18:43:29.692461 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:29.692470 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:29.692552 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:29.692658 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:29.694569 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:29.695919 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:29.697863 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:29.698595 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:29.698694 (+ 99us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:29.702340 (+ 3646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:29.702349 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:29.702355 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:29.702402 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:29.706128 (+ 3726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:29.707192 (+ 1064us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:29.707198 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:29.707202 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:29.707363 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:29.707957 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:29.707962 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:29.707966 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:29.708423 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:29.708723 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:29.709853 (+ 1130us) server_negotiation.cc:299] Negotiation successful
0620 18:43:29.710055 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 18:43:29.714166 12680 catalog_manager.cc:1347] Generated new cluster ID: 6cf25e77f8314a65a458aebe9bae59f5
I20260620 18:43:29.714275 12680 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:43:29.729310 12680 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:43:29.731129 12680 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:43:29.749305 12680 catalog_manager.cc:5868] T 00000000000000000000000000000000 P faaa4debfd134572aedc67bd4e1c0f3b: Generated new TSK 0
I20260620 18:43:29.750138 12680 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:43:29.762784 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 12602
I20260620 18:43:29.764925 12664 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:43:29.856957 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 12602
W20260620 18:43:29.870817 12587 connection.cc:535] client connection to 127.0.0.1:42277 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42277): Connection reset by peer (error 104)
2026-06-20T18:43:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1890 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980676253068-24012-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:43:29 dist-test-slave-d645 krb5kdc[12701](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:43:29 dist-test-slave-d645 krb5kdc[12701](info): set up 2 sockets
Jun 20 18:43:29 dist-test-slave-d645 krb5kdc[12701](info): commencing operation
krb5kdc: starting...
W20260620 18:43:30.879343 24012 mini_kdc.cc:121] Time spent starting KDC: real 0.990s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980676253068-24012-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980676253068-24012-0/krb5kdc/test-user.keytab.
Jun 20 18:43:31 dist-test-slave-d645 krb5kdc[12701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981011, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:43:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:43:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980676253068-24012-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980676253068-24012-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 18:43:31.124495 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980676253068-24012-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980676253068-24012-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980676253068-24012-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33409
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980676253068-24012-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33409
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:43:31.442829 12714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:43:31.443128 12714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:43:31.443266 12714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:43:31.450620 12714 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:43:31.450752 12714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:43:31.450836 12714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:43:31.450920 12714 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:43:31.451004 12714 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:43:31.460804 12714 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980676253068-24012-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980676253068-24012-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33409
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980676253068-24012-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33409
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980676253068-24012-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:43:31.462078 12714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:43:31.463351 12714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:43:31.475035 12719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:31.474993 12722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:31.475009 12720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:31.475860 12714 server_base.cc:1034] running on GCE node
I20260620 18:43:31.476802 12714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:43:31.478190 12714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:43:31.479398 12714 hybrid_clock.cc:648] HybridClock initialized: now 1781981011479358 us; error 31 us; skew 500 ppm
I20260620 18:43:31.479760 12714 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:43:31 dist-test-slave-d645 krb5kdc[12701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981011, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:43:31.482795 12714 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 18:43:31.484292 12714 webserver.cc:460] Webserver started at http://127.0.0.1:39055/ using document root <none> and password file <none>
I20260620 18:43:31.484907 12714 fs_manager.cc:359] Metadata directory not provided
I20260620 18:43:31.485018 12714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:43:31.485294 12714 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:43:31.488225 12714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980676253068-24012-0/minicluster-data/master-0/data/instance:
uuid: "bcb41a7ab3f14e18991527e438abba9b"
format_stamp: "Formatted at 2026-06-20 18:43:31 on dist-test-slave-d645"
server_key: "74375be219359771f5b24104286948cb"
server_key_iv: "c145f06032f516318a05be91af7a0669"
server_key_version: "encryptionkey@0"
I20260620 18:43:31.489001 12714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980676253068-24012-0/minicluster-data/master-0/wal/instance:
uuid: "bcb41a7ab3f14e18991527e438abba9b"
format_stamp: "Formatted at 2026-06-20 18:43:31 on dist-test-slave-d645"
server_key: "74375be219359771f5b24104286948cb"
server_key_iv: "c145f06032f516318a05be91af7a0669"
server_key_version: "encryptionkey@0"
I20260620 18:43:31.494055 12714 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 18:43:31.497298 12729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:31.498492 12714 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:43:31.498641 12714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980676253068-24012-0/minicluster-data/master-0/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980676253068-24012-0/minicluster-data/master-0/wal
uuid: "bcb41a7ab3f14e18991527e438abba9b"
format_stamp: "Formatted at 2026-06-20 18:43:31 on dist-test-slave-d645"
server_key: "74375be219359771f5b24104286948cb"
server_key_iv: "c145f06032f516318a05be91af7a0669"
server_key_version: "encryptionkey@0"
I20260620 18:43:31.498808 12714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980676253068-24012-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980676253068-24012-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980676253068-24012-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:43:31.518667 12714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:43:31.522719 12714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:43:31.523007 12714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:43:31.541114 12714 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33409
I20260620 18:43:31.541132 12780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33409 every 8 connection(s)
I20260620 18:43:31.542518 12714 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
I20260620 18:43:31.546001 12781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:43:31.547153 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 12714
I20260620 18:43:31.547335 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980676253068-24012-0/minicluster-data/master-0/wal/instance
I20260620 18:43:31.547884 24012 external_mini_cluster.cc:1428] Setting key 5e1d71c8331fbd5bdf986b2e024362e1
I20260620 18:43:31.558185 12781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bcb41a7ab3f14e18991527e438abba9b: Bootstrap starting.
I20260620 18:43:31.560070 12784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:31.549738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.21:50235 (local address 127.0.0.1:33409)
0620 18:43:31.550555 (+ 817us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:31.550575 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:31.550620 (+ 45us) server_negotiation.cc:407] Connection header received
0620 18:43:31.552267 (+ 1647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:31.552287 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:31.552676 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:31.553015 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:31.553506 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:31.554712 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:31.555970 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:31.556638 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:31.556802 (+ 164us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:31.557930 (+ 1128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:31.557937 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:31.557950 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:43:31.558499 (+ 549us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:31.558573 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:43:31.558602 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:31.558727 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:31.559013 (+ 286us) server_negotiation.cc:299] Negotiation successful
0620 18:43:31.559161 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":359,"threads_started":1}
I20260620 18:43:31.561342 12781 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bcb41a7ab3f14e18991527e438abba9b: Neither blocks nor log segments found. Creating new log.
I20260620 18:43:31.562386 12781 log.cc:824] T 00000000000000000000000000000000 P bcb41a7ab3f14e18991527e438abba9b: Log is configured to *not* fsync() on all Append() calls
I20260620 18:43:31.563709 24012 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:43:31.565470 12781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bcb41a7ab3f14e18991527e438abba9b: No bootstrap required, opened a new log
I20260620 18:43:31.576007 12784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:31.568791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.21:38835 (local address 127.0.0.1:33409)
0620 18:43:31.568975 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:31.568981 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:31.569425 (+ 444us) server_negotiation.cc:407] Connection header received
0620 18:43:31.569590 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:31.569596 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:31.569676 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:31.569790 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:31.570280 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:31.571458 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:31.573009 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:31.573727 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:31.573869 (+ 142us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:31.575178 (+ 1309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:31.575186 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:31.575190 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:43:31.575211 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:31.575274 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:43:31.575295 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:31.575422 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:31.575642 (+ 220us) server_negotiation.cc:299] Negotiation successful
0620 18:43:31.575697 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 18:43:31.576041 12781 raft_consensus.cc:348] T 00000000000000000000000000000000 P bcb41a7ab3f14e18991527e438abba9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcb41a7ab3f14e18991527e438abba9b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33409 } }
I20260620 18:43:31.576571 12781 raft_consensus.cc:374] T 00000000000000000000000000000000 P bcb41a7ab3f14e18991527e438abba9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:43:31.576694 12781 raft_consensus.cc:729] T 00000000000000000000000000000000 P bcb41a7ab3f14e18991527e438abba9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcb41a7ab3f14e18991527e438abba9b, State: Initialized, Role: FOLLOWER
I20260620 18:43:31.577874 12781 consensus_queue.cc:260] T 00000000000000000000000000000000 P bcb41a7ab3f14e18991527e438abba9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcb41a7ab3f14e18991527e438abba9b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33409 } }
I20260620 18:43:31.578173 12781 raft_consensus.cc:388] T 00000000000000000000000000000000 P bcb41a7ab3f14e18991527e438abba9b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:43:31.578306 12781 raft_consensus.cc:482] T 00000000000000000000000000000000 P bcb41a7ab3f14e18991527e438abba9b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:43:31.578459 12781 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bcb41a7ab3f14e18991527e438abba9b [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:43:31.580782 12781 raft_consensus.cc:504] T 00000000000000000000000000000000 P bcb41a7ab3f14e18991527e438abba9b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcb41a7ab3f14e18991527e438abba9b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33409 } }
I20260620 18:43:31.581315 12781 leader_election.cc:302] T 00000000000000000000000000000000 P bcb41a7ab3f14e18991527e438abba9b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bcb41a7ab3f14e18991527e438abba9b; no voters:
I20260620 18:43:31.581969 12781 leader_election.cc:288] T 00000000000000000000000000000000 P bcb41a7ab3f14e18991527e438abba9b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:43:31.582099 12791 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bcb41a7ab3f14e18991527e438abba9b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:43:31.582733 12791 raft_consensus.cc:686] T 00000000000000000000000000000000 P bcb41a7ab3f14e18991527e438abba9b [term 1 LEADER]: Becoming Leader. State: Replica: bcb41a7ab3f14e18991527e438abba9b, State: Running, Role: LEADER
I20260620 18:43:31.583782 12791 consensus_queue.cc:237] T 00000000000000000000000000000000 P bcb41a7ab3f14e18991527e438abba9b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcb41a7ab3f14e18991527e438abba9b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33409 } }
I20260620 18:43:31.584211 12781 sys_catalog.cc:564] T 00000000000000000000000000000000 P bcb41a7ab3f14e18991527e438abba9b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:43:31.587081 12792 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcb41a7ab3f14e18991527e438abba9b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bcb41a7ab3f14e18991527e438abba9b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcb41a7ab3f14e18991527e438abba9b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33409 } } }
I20260620 18:43:31.587466 12792 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcb41a7ab3f14e18991527e438abba9b [sys.catalog]: This master's current role is: LEADER
I20260620 18:43:31.587356 12793 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcb41a7ab3f14e18991527e438abba9b [sys.catalog]: SysCatalogTable state changed. Reason: New leader bcb41a7ab3f14e18991527e438abba9b. Latest consensus state: current_term: 1 leader_uuid: "bcb41a7ab3f14e18991527e438abba9b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcb41a7ab3f14e18991527e438abba9b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33409 } } }
I20260620 18:43:31.587622 12793 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcb41a7ab3f14e18991527e438abba9b [sys.catalog]: This master's current role is: LEADER
I20260620 18:43:31.588249 12800 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:43:31.594326 12800 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:43:31.603576 12800 catalog_manager.cc:1347] Generated new cluster ID: c2d8933140f0410a81bf80d6e2a3ad7f
I20260620 18:43:31.603704 12800 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:43:31.614972 12800 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:43:31.616847 12800 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:43:31.624729 12800 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bcb41a7ab3f14e18991527e438abba9b: Generated new TSK 0
I20260620 18:43:31.626194 12800 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:43:31.635222 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 12714
W20260620 18:43:31.636857 12776 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.21:50235
W20260620 18:43:31.637981 24012 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 12714: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:43:31.638195 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 12714
W20260620 18:43:31.652827 12699 connection.cc:535] client connection to 127.0.0.1:33409 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33409): Connection reset by peer (error 104)
2026-06-20T18:43:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1782 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980676253068-24012-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:43:31 dist-test-slave-d645 krb5kdc[12812](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:43:31 dist-test-slave-d645 krb5kdc[12812](info): set up 2 sockets
Jun 20 18:43:31 dist-test-slave-d645 krb5kdc[12812](info): commencing operation
krb5kdc: starting...
W20260620 18:43:32.663405 24012 mini_kdc.cc:121] Time spent starting KDC: real 0.991s user 0.008s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980676253068-24012-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980676253068-24012-0/krb5kdc/test-user.keytab.
Jun 20 18:43:32 dist-test-slave-d645 krb5kdc[12812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981012, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:43:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:43:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980676253068-24012-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980676253068-24012-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 18:43:32.906196 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980676253068-24012-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980676253068-24012-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980676253068-24012-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35253
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980676253068-24012-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35253
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:43:33.240633 12825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:43:33.240913 12825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:43:33.241041 12825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:43:33.248675 12825 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:43:33.248801 12825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:43:33.248860 12825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:43:33.248943 12825 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:43:33.249003 12825 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:43:33.259400 12825 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980676253068-24012-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980676253068-24012-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35253
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980676253068-24012-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980676253068-24012-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:43:33.260648 12825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:43:33.261899 12825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:43:33.273478 12833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:33.273568 12830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:33.273527 12831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:33.274501 12825 server_base.cc:1034] running on GCE node
I20260620 18:43:33.275267 12825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:43:33.276587 12825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:43:33.277798 12825 hybrid_clock.cc:648] HybridClock initialized: now 1781981013277758 us; error 32 us; skew 500 ppm
I20260620 18:43:33.278148 12825 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:43:33 dist-test-slave-d645 krb5kdc[12812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981013, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:43:33.285480 12825 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 18:43:33.287115 12825 webserver.cc:460] Webserver started at http://127.0.0.1:40991/ using document root <none> and password file <none>
I20260620 18:43:33.287773 12825 fs_manager.cc:359] Metadata directory not provided
I20260620 18:43:33.287875 12825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:43:33.288161 12825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:43:33.291260 12825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980676253068-24012-0/minicluster-data/master-0/data/instance:
uuid: "76f69248e0a14caa82f7493adab38afa"
format_stamp: "Formatted at 2026-06-20 18:43:33 on dist-test-slave-d645"
server_key: "6f16890ad08ab0c84787ef99ca98dc67"
server_key_iv: "02c00bd8d7eec392790360d43f1b093a"
server_key_version: "encryptionkey@0"
I20260620 18:43:33.292052 12825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980676253068-24012-0/minicluster-data/master-0/wal/instance:
uuid: "76f69248e0a14caa82f7493adab38afa"
format_stamp: "Formatted at 2026-06-20 18:43:33 on dist-test-slave-d645"
server_key: "6f16890ad08ab0c84787ef99ca98dc67"
server_key_iv: "02c00bd8d7eec392790360d43f1b093a"
server_key_version: "encryptionkey@0"
I20260620 18:43:33.297189 12825 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:43:33.300673 12840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:33.302011 12825 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 18:43:33.302168 12825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980676253068-24012-0/minicluster-data/master-0/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980676253068-24012-0/minicluster-data/master-0/wal
uuid: "76f69248e0a14caa82f7493adab38afa"
format_stamp: "Formatted at 2026-06-20 18:43:33 on dist-test-slave-d645"
server_key: "6f16890ad08ab0c84787ef99ca98dc67"
server_key_iv: "02c00bd8d7eec392790360d43f1b093a"
server_key_version: "encryptionkey@0"
I20260620 18:43:33.302340 12825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980676253068-24012-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980676253068-24012-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980676253068-24012-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:43:33.308151 12825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:43:33.312122 12825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:43:33.312355 12825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:43:33.328853 12825 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35253
I20260620 18:43:33.328896 12891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35253 every 8 connection(s)
I20260620 18:43:33.330163 12825 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
I20260620 18:43:33.333777 12892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:43:33.339846 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 12825
I20260620 18:43:33.340036 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980676253068-24012-0/minicluster-data/master-0/wal/instance
I20260620 18:43:33.340584 24012 external_mini_cluster.cc:1428] Setting key 453ca320faa09ae26dadc5b3e0b2f64d
I20260620 18:43:33.344739 12892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76f69248e0a14caa82f7493adab38afa: Bootstrap starting.
I20260620 18:43:33.348506 12892 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 76f69248e0a14caa82f7493adab38afa: Neither blocks nor log segments found. Creating new log.
I20260620 18:43:33.348482 12895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:33.342394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.21:43643 (local address 127.0.0.1:35253)
0620 18:43:33.343223 (+ 829us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:33.343235 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:33.343275 (+ 40us) server_negotiation.cc:407] Connection header received
0620 18:43:33.344887 (+ 1612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:33.344915 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:33.345260 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:33.345563 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:33.346713 (+ 1150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:33.346720 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:33.346732 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:43:33.346753 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:33.346826 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:43:33.346848 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:33.347089 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:33.347188 (+ 99us) server_negotiation.cc:299] Negotiation successful
0620 18:43:33.347370 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":362,"threads_started":1}
I20260620 18:43:33.350154 12892 log.cc:824] T 00000000000000000000000000000000 P 76f69248e0a14caa82f7493adab38afa: Log is configured to *not* fsync() on all Append() calls
I20260620 18:43:33.352949 24012 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:43:33.353274 12892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76f69248e0a14caa82f7493adab38afa: No bootstrap required, opened a new log
I20260620 18:43:33.359814 12895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:33.357092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.21:53565 (local address 127.0.0.1:35253)
0620 18:43:33.357292 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:33.357298 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:33.357625 (+ 327us) server_negotiation.cc:407] Connection header received
0620 18:43:33.357828 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:33.357832 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:33.357891 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:33.357985 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:33.358979 (+ 994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:33.358985 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:33.358989 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:43:33.359006 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:33.359048 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:43:33.359135 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:33.359285 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:33.359428 (+ 143us) server_negotiation.cc:299] Negotiation successful
0620 18:43:33.359506 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 18:43:33.361588 12892 raft_consensus.cc:348] T 00000000000000000000000000000000 P 76f69248e0a14caa82f7493adab38afa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f69248e0a14caa82f7493adab38afa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35253 } }
I20260620 18:43:33.361951 12892 raft_consensus.cc:374] T 00000000000000000000000000000000 P 76f69248e0a14caa82f7493adab38afa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:43:33.362037 12892 raft_consensus.cc:729] T 00000000000000000000000000000000 P 76f69248e0a14caa82f7493adab38afa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76f69248e0a14caa82f7493adab38afa, State: Initialized, Role: FOLLOWER
I20260620 18:43:33.362589 12892 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76f69248e0a14caa82f7493adab38afa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f69248e0a14caa82f7493adab38afa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35253 } }
I20260620 18:43:33.362815 12892 raft_consensus.cc:388] T 00000000000000000000000000000000 P 76f69248e0a14caa82f7493adab38afa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:43:33.362984 12892 raft_consensus.cc:482] T 00000000000000000000000000000000 P 76f69248e0a14caa82f7493adab38afa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:43:33.363147 12892 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 76f69248e0a14caa82f7493adab38afa [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:43:33.365269 12892 raft_consensus.cc:504] T 00000000000000000000000000000000 P 76f69248e0a14caa82f7493adab38afa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f69248e0a14caa82f7493adab38afa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35253 } }
I20260620 18:43:33.365734 12892 leader_election.cc:302] T 00000000000000000000000000000000 P 76f69248e0a14caa82f7493adab38afa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76f69248e0a14caa82f7493adab38afa; no voters:
I20260620 18:43:33.366338 12892 leader_election.cc:288] T 00000000000000000000000000000000 P 76f69248e0a14caa82f7493adab38afa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:43:33.366430 12902 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 76f69248e0a14caa82f7493adab38afa [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:43:33.367153 12902 raft_consensus.cc:686] T 00000000000000000000000000000000 P 76f69248e0a14caa82f7493adab38afa [term 1 LEADER]: Becoming Leader. State: Replica: 76f69248e0a14caa82f7493adab38afa, State: Running, Role: LEADER
I20260620 18:43:33.367625 12902 consensus_queue.cc:237] T 00000000000000000000000000000000 P 76f69248e0a14caa82f7493adab38afa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f69248e0a14caa82f7493adab38afa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35253 } }
I20260620 18:43:33.368379 12892 sys_catalog.cc:564] T 00000000000000000000000000000000 P 76f69248e0a14caa82f7493adab38afa [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:43:33.370172 12904 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76f69248e0a14caa82f7493adab38afa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76f69248e0a14caa82f7493adab38afa. Latest consensus state: current_term: 1 leader_uuid: "76f69248e0a14caa82f7493adab38afa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f69248e0a14caa82f7493adab38afa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35253 } } }
I20260620 18:43:33.370615 12904 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76f69248e0a14caa82f7493adab38afa [sys.catalog]: This master's current role is: LEADER
I20260620 18:43:33.371192 12903 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76f69248e0a14caa82f7493adab38afa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "76f69248e0a14caa82f7493adab38afa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f69248e0a14caa82f7493adab38afa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35253 } } }
I20260620 18:43:33.371495 12903 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76f69248e0a14caa82f7493adab38afa [sys.catalog]: This master's current role is: LEADER
I20260620 18:43:33.372906 12911 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:43:33.378089 12911 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:43:33.386739 12911 catalog_manager.cc:1347] Generated new cluster ID: 7569326a47e5422a99d8005542307833
I20260620 18:43:33.386852 12911 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:43:33.400653 12911 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:43:33.401134 12911 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:43:33.413561 12911 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 76f69248e0a14caa82f7493adab38afa: Generated new TSK 0
I20260620 18:43:33.414592 12911 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:43:33.421171 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 12825
W20260620 18:43:33.422231 12887 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.21:43643
W20260620 18:43:33.422876 24012 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 12825: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:43:33.423053 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 12825
2026-06-20T18:43:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1783 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980676253068-24012-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:43:33 dist-test-slave-d645 krb5kdc[12923](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:43:33 dist-test-slave-d645 krb5kdc[12923](info): set up 2 sockets
Jun 20 18:43:33 dist-test-slave-d645 krb5kdc[12923](info): commencing operation
krb5kdc: starting...
W20260620 18:43:34.443403 24012 mini_kdc.cc:121] Time spent starting KDC: real 0.988s user 0.006s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980676253068-24012-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980676253068-24012-0/krb5kdc/test-user.keytab.
Jun 20 18:43:34 dist-test-slave-d645 krb5kdc[12923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:43:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:43:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980676253068-24012-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980676253068-24012-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 18:43:34.676343 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980676253068-24012-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980676253068-24012-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980676253068-24012-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39849
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980676253068-24012-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39849
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:43:34.996430 12936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:43:34.996753 12936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:43:34.996846 12936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:43:35.004359 12936 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:43:35.004487 12936 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:43:35.004530 12936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:43:35.004590 12936 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:43:35.004631 12936 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:43:35.015973 12936 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980676253068-24012-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980676253068-24012-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39849
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980676253068-24012-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39849
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980676253068-24012-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:43:35.017974 12936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:43:35.019953 12936 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:43:35.035440 12942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:35.035425 12944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:35.035681 12941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:35.036652 12936 server_base.cc:1034] running on GCE node
I20260620 18:43:35.037603 12936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:43:35.039083 12936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:43:35.040294 12936 hybrid_clock.cc:648] HybridClock initialized: now 1781981015040251 us; error 34 us; skew 500 ppm
I20260620 18:43:35.040647 12936 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:43:35 dist-test-slave-d645 krb5kdc[12923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981015, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:43:35.043773 12936 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 18:43:35.045214 12936 webserver.cc:460] Webserver started at http://127.0.0.1:43005/ using document root <none> and password file <none>
I20260620 18:43:35.045897 12936 fs_manager.cc:359] Metadata directory not provided
I20260620 18:43:35.046000 12936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:43:35.046299 12936 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:43:35.049273 12936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980676253068-24012-0/minicluster-data/master-0/data/instance:
uuid: "c272ea0cd9e640e6b11e05bff5847bf0"
format_stamp: "Formatted at 2026-06-20 18:43:35 on dist-test-slave-d645"
server_key: "d9791f493582326717d6454375baab66"
server_key_iv: "adc9952d2234c4e15ca2a8ccab7576a5"
server_key_version: "encryptionkey@0"
I20260620 18:43:35.050019 12936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980676253068-24012-0/minicluster-data/master-0/wal/instance:
uuid: "c272ea0cd9e640e6b11e05bff5847bf0"
format_stamp: "Formatted at 2026-06-20 18:43:35 on dist-test-slave-d645"
server_key: "d9791f493582326717d6454375baab66"
server_key_iv: "adc9952d2234c4e15ca2a8ccab7576a5"
server_key_version: "encryptionkey@0"
I20260620 18:43:35.054976 12936 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.006s
I20260620 18:43:35.058413 12951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:35.059713 12936 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 18:43:35.059864 12936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980676253068-24012-0/minicluster-data/master-0/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980676253068-24012-0/minicluster-data/master-0/wal
uuid: "c272ea0cd9e640e6b11e05bff5847bf0"
format_stamp: "Formatted at 2026-06-20 18:43:35 on dist-test-slave-d645"
server_key: "d9791f493582326717d6454375baab66"
server_key_iv: "adc9952d2234c4e15ca2a8ccab7576a5"
server_key_version: "encryptionkey@0"
I20260620 18:43:35.060032 12936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980676253068-24012-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980676253068-24012-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980676253068-24012-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:43:35.078632 12936 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:43:35.082448 12936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:43:35.082712 12936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:43:35.101042 12936 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39849
I20260620 18:43:35.101063 13002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39849 every 8 connection(s)
I20260620 18:43:35.102398 12936 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
I20260620 18:43:35.106320 13003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:43:35.109407 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 12936
I20260620 18:43:35.109598 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980676253068-24012-0/minicluster-data/master-0/wal/instance
I20260620 18:43:35.110133 24012 external_mini_cluster.cc:1428] Setting key f35335631fa8184d3dfc6f695f90814c
I20260620 18:43:35.118300 13003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c272ea0cd9e640e6b11e05bff5847bf0: Bootstrap starting.
I20260620 18:43:35.121903 13003 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c272ea0cd9e640e6b11e05bff5847bf0: Neither blocks nor log segments found. Creating new log.
I20260620 18:43:35.123183 13003 log.cc:824] T 00000000000000000000000000000000 P c272ea0cd9e640e6b11e05bff5847bf0: Log is configured to *not* fsync() on all Append() calls
Jun 20 18:43:35 dist-test-slave-d645 krb5kdc[12923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 18:43:35.126302 13003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c272ea0cd9e640e6b11e05bff5847bf0: No bootstrap required, opened a new log
I20260620 18:43:35.132771 13006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:35.111937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.21:54509 (local address 127.0.0.1:39849)
0620 18:43:35.112634 (+ 697us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:35.112645 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:35.112685 (+ 40us) server_negotiation.cc:407] Connection header received
0620 18:43:35.114314 (+ 1629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:35.114336 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:35.114715 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:35.115054 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:35.116817 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:35.118074 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:35.119809 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:35.120473 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:35.120595 (+ 122us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:35.125481 (+ 4886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:35.125489 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:35.125504 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:35.125543 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:35.128430 (+ 2887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:35.129438 (+ 1008us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:35.129444 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:35.129450 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:35.129634 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:35.130085 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:35.130089 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:35.130091 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:35.130594 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:35.130844 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:35.131354 (+ 510us) server_negotiation.cc:299] Negotiation successful
0620 18:43:35.131667 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":307,"threads_started":1}
I20260620 18:43:35.135015 13003 raft_consensus.cc:348] T 00000000000000000000000000000000 P c272ea0cd9e640e6b11e05bff5847bf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c272ea0cd9e640e6b11e05bff5847bf0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39849 } }
I20260620 18:43:35.135428 13003 raft_consensus.cc:374] T 00000000000000000000000000000000 P c272ea0cd9e640e6b11e05bff5847bf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:43:35.135526 13003 raft_consensus.cc:729] T 00000000000000000000000000000000 P c272ea0cd9e640e6b11e05bff5847bf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c272ea0cd9e640e6b11e05bff5847bf0, State: Initialized, Role: FOLLOWER
I20260620 18:43:35.136085 13003 consensus_queue.cc:260] T 00000000000000000000000000000000 P c272ea0cd9e640e6b11e05bff5847bf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c272ea0cd9e640e6b11e05bff5847bf0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39849 } }
I20260620 18:43:35.136309 13003 raft_consensus.cc:388] T 00000000000000000000000000000000 P c272ea0cd9e640e6b11e05bff5847bf0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:43:35.136439 13003 raft_consensus.cc:482] T 00000000000000000000000000000000 P c272ea0cd9e640e6b11e05bff5847bf0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:43:35.136509 24012 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:43:35.136587 13003 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c272ea0cd9e640e6b11e05bff5847bf0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:43:35.138621 13003 raft_consensus.cc:504] T 00000000000000000000000000000000 P c272ea0cd9e640e6b11e05bff5847bf0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c272ea0cd9e640e6b11e05bff5847bf0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39849 } }
I20260620 18:43:35.139317 13003 leader_election.cc:302] T 00000000000000000000000000000000 P c272ea0cd9e640e6b11e05bff5847bf0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c272ea0cd9e640e6b11e05bff5847bf0; no voters:
I20260620 18:43:35.140118 13003 leader_election.cc:288] T 00000000000000000000000000000000 P c272ea0cd9e640e6b11e05bff5847bf0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:43:35.140246 13012 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c272ea0cd9e640e6b11e05bff5847bf0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:43:35.142237 13012 raft_consensus.cc:686] T 00000000000000000000000000000000 P c272ea0cd9e640e6b11e05bff5847bf0 [term 1 LEADER]: Becoming Leader. State: Replica: c272ea0cd9e640e6b11e05bff5847bf0, State: Running, Role: LEADER
I20260620 18:43:35.142490 13003 sys_catalog.cc:564] T 00000000000000000000000000000000 P c272ea0cd9e640e6b11e05bff5847bf0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:43:35.142874 13012 consensus_queue.cc:237] T 00000000000000000000000000000000 P c272ea0cd9e640e6b11e05bff5847bf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c272ea0cd9e640e6b11e05bff5847bf0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39849 } }
I20260620 18:43:35.146972 13013 sys_catalog.cc:455] T 00000000000000000000000000000000 P c272ea0cd9e640e6b11e05bff5847bf0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c272ea0cd9e640e6b11e05bff5847bf0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c272ea0cd9e640e6b11e05bff5847bf0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39849 } } }
I20260620 18:43:35.146869 13014 sys_catalog.cc:455] T 00000000000000000000000000000000 P c272ea0cd9e640e6b11e05bff5847bf0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c272ea0cd9e640e6b11e05bff5847bf0. Latest consensus state: current_term: 1 leader_uuid: "c272ea0cd9e640e6b11e05bff5847bf0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c272ea0cd9e640e6b11e05bff5847bf0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39849 } } }
I20260620 18:43:35.147248 13014 sys_catalog.cc:458] T 00000000000000000000000000000000 P c272ea0cd9e640e6b11e05bff5847bf0 [sys.catalog]: This master's current role is: LEADER
I20260620 18:43:35.147248 13013 sys_catalog.cc:458] T 00000000000000000000000000000000 P c272ea0cd9e640e6b11e05bff5847bf0 [sys.catalog]: This master's current role is: LEADER
I20260620 18:43:35.159901 13022 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:43:35.159992 13006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:35.141680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.21:41567 (local address 127.0.0.1:39849)
0620 18:43:35.141844 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:35.141850 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:35.141868 (+ 18us) server_negotiation.cc:407] Connection header received
0620 18:43:35.142070 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:35.142076 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:35.142171 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:35.142269 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:35.144260 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:35.145520 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:35.147399 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:35.148193 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:35.148303 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:35.151868 (+ 3565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:35.151876 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:35.151882 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:35.151922 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:35.155437 (+ 3515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:35.156461 (+ 1024us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:35.156467 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:35.156470 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:35.156648 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:35.157223 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:35.157229 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:35.157233 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:35.157644 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:35.157914 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:35.159268 (+ 1354us) server_negotiation.cc:299] Negotiation successful
0620 18:43:35.159561 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 18:43:35.165314 13022 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:43:35.173538 13022 catalog_manager.cc:1347] Generated new cluster ID: 88526853b42e48f48d50827d288ce3a7
I20260620 18:43:35.173662 13022 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:43:35.185006 13022 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:43:35.186738 13022 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:43:35.204571 13022 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c272ea0cd9e640e6b11e05bff5847bf0: Generated new TSK 0
I20260620 18:43:35.205624 13022 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:43:35.216522 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 12936
I20260620 18:43:35.218535 12998 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:43:35.308895 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 12936
W20260620 18:43:35.323320 12921 connection.cc:535] client connection to 127.0.0.1:39849 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39849): Connection reset by peer (error 104)
2026-06-20T18:43:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1886 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980676253068-24012-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:43:35 dist-test-slave-d645 krb5kdc[13035](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:43:35 dist-test-slave-d645 krb5kdc[13035](info): set up 2 sockets
Jun 20 18:43:35 dist-test-slave-d645 krb5kdc[13035](info): commencing operation
krb5kdc: starting...
W20260620 18:43:36.331512 24012 mini_kdc.cc:121] Time spent starting KDC: real 0.990s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980676253068-24012-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980676253068-24012-0/krb5kdc/test-user.keytab.
Jun 20 18:43:36 dist-test-slave-d645 krb5kdc[13035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981016, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:43:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:43:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980676253068-24012-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980676253068-24012-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 18:43:36.578817 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980676253068-24012-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980676253068-24012-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980676253068-24012-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40539
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980676253068-24012-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40539
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:43:36.907678 13048 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:43:36.907985 13048 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:43:36.908126 13048 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:43:36.915452 13048 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:43:36.915582 13048 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:43:36.915669 13048 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:43:36.915753 13048 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:43:36.915812 13048 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:43:36.925853 13048 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980676253068-24012-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980676253068-24012-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40539
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980676253068-24012-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980676253068-24012-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:43:36.927896 13048 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:43:36.929301 13048 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:43:36.940842 13053 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:36.940805 13054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:36.941354 13056 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:36.941406 13048 server_base.cc:1034] running on GCE node
I20260620 18:43:36.942174 13048 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:43:36.943586 13048 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:43:36.944813 13048 hybrid_clock.cc:648] HybridClock initialized: now 1781981016944765 us; error 42 us; skew 500 ppm
I20260620 18:43:36.945142 13048 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:43:36 dist-test-slave-d645 krb5kdc[13035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981016, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:43:36.948383 13048 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 18:43:36.949852 13048 webserver.cc:460] Webserver started at http://127.0.0.1:41289/ using document root <none> and password file <none>
I20260620 18:43:36.950487 13048 fs_manager.cc:359] Metadata directory not provided
I20260620 18:43:36.950604 13048 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:43:36.950886 13048 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:43:36.953785 13048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980676253068-24012-0/minicluster-data/master-0/data/instance:
uuid: "fa87ba25eee94814985f265b2619bd04"
format_stamp: "Formatted at 2026-06-20 18:43:36 on dist-test-slave-d645"
server_key: "682e2d25d22ad4353a3d2f2a931e55e0"
server_key_iv: "eac29d27dae342558de3c5723b0b8250"
server_key_version: "encryptionkey@0"
I20260620 18:43:36.954495 13048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980676253068-24012-0/minicluster-data/master-0/wal/instance:
uuid: "fa87ba25eee94814985f265b2619bd04"
format_stamp: "Formatted at 2026-06-20 18:43:36 on dist-test-slave-d645"
server_key: "682e2d25d22ad4353a3d2f2a931e55e0"
server_key_iv: "eac29d27dae342558de3c5723b0b8250"
server_key_version: "encryptionkey@0"
I20260620 18:43:36.959832 13048 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260620 18:43:36.963306 13063 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:36.964509 13048 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:43:36.964668 13048 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980676253068-24012-0/minicluster-data/master-0/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980676253068-24012-0/minicluster-data/master-0/wal
uuid: "fa87ba25eee94814985f265b2619bd04"
format_stamp: "Formatted at 2026-06-20 18:43:36 on dist-test-slave-d645"
server_key: "682e2d25d22ad4353a3d2f2a931e55e0"
server_key_iv: "eac29d27dae342558de3c5723b0b8250"
server_key_version: "encryptionkey@0"
I20260620 18:43:36.964838 13048 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980676253068-24012-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980676253068-24012-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980676253068-24012-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:43:36.980130 13048 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:43:36.984216 13048 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:43:36.984473 13048 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:43:37.002158 13048 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40539
I20260620 18:43:37.002225 13114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40539 every 8 connection(s)
I20260620 18:43:37.003734 13048 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
I20260620 18:43:37.004144 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 13048
I20260620 18:43:37.004312 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980676253068-24012-0/minicluster-data/master-0/wal/instance
I20260620 18:43:37.004832 24012 external_mini_cluster.cc:1428] Setting key 4204070ff800fe1f10170500b9347fca
I20260620 18:43:37.008155 13115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:43:37.022060 13118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:37.006803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.21:55817 (local address 127.0.0.1:40539)
0620 18:43:37.008282 (+ 1479us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:37.008298 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:37.008346 (+ 48us) server_negotiation.cc:407] Connection header received
0620 18:43:37.010839 (+ 2493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:37.010866 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:37.011413 (+ 547us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:37.011856 (+ 443us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:37.012377 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:37.014352 (+ 1975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:37.015956 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:37.016964 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:37.017172 (+ 208us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:37.018914 (+ 1742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:37.018924 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:37.018967 (+ 43us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:43:37.019738 (+ 771us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:37.019837 (+ 99us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:43:37.019884 (+ 47us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:37.020156 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:37.020558 (+ 402us) server_negotiation.cc:299] Negotiation successful
0620 18:43:37.020756 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1206,"thread_start_us":851,"threads_started":1}
I20260620 18:43:37.022615 13115 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa87ba25eee94814985f265b2619bd04: Bootstrap starting.
I20260620 18:43:37.026188 13115 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fa87ba25eee94814985f265b2619bd04: Neither blocks nor log segments found. Creating new log.
I20260620 18:43:37.026358 24012 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:43:37.027227 13115 log.cc:824] T 00000000000000000000000000000000 P fa87ba25eee94814985f265b2619bd04: Log is configured to *not* fsync() on all Append() calls
I20260620 18:43:37.030052 13115 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa87ba25eee94814985f265b2619bd04: No bootstrap required, opened a new log
I20260620 18:43:37.037410 13118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:37.030741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.21:59219 (local address 127.0.0.1:40539)
0620 18:43:37.030915 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:37.030921 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:37.031341 (+ 420us) server_negotiation.cc:407] Connection header received
0620 18:43:37.031478 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:37.031484 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:37.031569 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:37.031696 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:37.032130 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:37.033313 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:37.034650 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:37.035387 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:37.035518 (+ 131us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:37.036576 (+ 1058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:37.036583 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:37.036588 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:43:37.036610 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:37.036661 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:43:37.036680 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:37.036789 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:37.037007 (+ 218us) server_negotiation.cc:299] Negotiation successful
0620 18:43:37.037064 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 18:43:37.039433 13115 raft_consensus.cc:348] T 00000000000000000000000000000000 P fa87ba25eee94814985f265b2619bd04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa87ba25eee94814985f265b2619bd04" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40539 } }
I20260620 18:43:37.039788 13115 raft_consensus.cc:374] T 00000000000000000000000000000000 P fa87ba25eee94814985f265b2619bd04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:43:37.039862 13115 raft_consensus.cc:729] T 00000000000000000000000000000000 P fa87ba25eee94814985f265b2619bd04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa87ba25eee94814985f265b2619bd04, State: Initialized, Role: FOLLOWER
I20260620 18:43:37.040395 13115 consensus_queue.cc:260] T 00000000000000000000000000000000 P fa87ba25eee94814985f265b2619bd04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa87ba25eee94814985f265b2619bd04" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40539 } }
I20260620 18:43:37.040616 13115 raft_consensus.cc:388] T 00000000000000000000000000000000 P fa87ba25eee94814985f265b2619bd04 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:43:37.040719 13115 raft_consensus.cc:482] T 00000000000000000000000000000000 P fa87ba25eee94814985f265b2619bd04 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:43:37.040838 13115 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fa87ba25eee94814985f265b2619bd04 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:43:37.043239 13115 raft_consensus.cc:504] T 00000000000000000000000000000000 P fa87ba25eee94814985f265b2619bd04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa87ba25eee94814985f265b2619bd04" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40539 } }
I20260620 18:43:37.043677 13115 leader_election.cc:302] T 00000000000000000000000000000000 P fa87ba25eee94814985f265b2619bd04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fa87ba25eee94814985f265b2619bd04; no voters:
I20260620 18:43:37.044183 13115 leader_election.cc:288] T 00000000000000000000000000000000 P fa87ba25eee94814985f265b2619bd04 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:43:37.044302 13125 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fa87ba25eee94814985f265b2619bd04 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:43:37.044937 13125 raft_consensus.cc:686] T 00000000000000000000000000000000 P fa87ba25eee94814985f265b2619bd04 [term 1 LEADER]: Becoming Leader. State: Replica: fa87ba25eee94814985f265b2619bd04, State: Running, Role: LEADER
I20260620 18:43:37.045472 13125 consensus_queue.cc:237] T 00000000000000000000000000000000 P fa87ba25eee94814985f265b2619bd04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa87ba25eee94814985f265b2619bd04" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40539 } }
I20260620 18:43:37.046104 13115 sys_catalog.cc:564] T 00000000000000000000000000000000 P fa87ba25eee94814985f265b2619bd04 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:43:37.048187 13126 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa87ba25eee94814985f265b2619bd04 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fa87ba25eee94814985f265b2619bd04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa87ba25eee94814985f265b2619bd04" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40539 } } }
I20260620 18:43:37.048218 13127 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa87ba25eee94814985f265b2619bd04 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fa87ba25eee94814985f265b2619bd04. Latest consensus state: current_term: 1 leader_uuid: "fa87ba25eee94814985f265b2619bd04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa87ba25eee94814985f265b2619bd04" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40539 } } }
I20260620 18:43:37.048575 13126 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa87ba25eee94814985f265b2619bd04 [sys.catalog]: This master's current role is: LEADER
I20260620 18:43:37.048575 13127 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa87ba25eee94814985f265b2619bd04 [sys.catalog]: This master's current role is: LEADER
I20260620 18:43:37.054217 13132 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:43:37.059417 13132 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:43:37.068241 13132 catalog_manager.cc:1347] Generated new cluster ID: 5ba226dbeab64fd9b17091993bd33f1f
I20260620 18:43:37.068346 13132 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:43:37.099417 13132 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:43:37.101670 13132 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:43:37.112299 13132 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fa87ba25eee94814985f265b2619bd04: Generated new TSK 0
I20260620 18:43:37.113379 13132 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:43:37.130232 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 13048
W20260620 18:43:37.131564 13110 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.21:55817
W20260620 18:43:37.132282 24012 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 13048: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:43:37.132400 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 13048
W20260620 18:43:37.146340 13033 connection.cc:535] client connection to 127.0.0.1:40539 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40539): Connection reset by peer (error 104)
2026-06-20T18:43:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1823 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980676253068-24012-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:43:37 dist-test-slave-d645 krb5kdc[13146](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:43:37 dist-test-slave-d645 krb5kdc[13146](info): set up 2 sockets
Jun 20 18:43:37 dist-test-slave-d645 krb5kdc[13146](info): commencing operation
krb5kdc: starting...
W20260620 18:43:38.159605 24012 mini_kdc.cc:121] Time spent starting KDC: real 0.994s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980676253068-24012-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980676253068-24012-0/krb5kdc/test-user.keytab.
Jun 20 18:43:38 dist-test-slave-d645 krb5kdc[13146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981018, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:43:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:43:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980676253068-24012-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980676253068-24012-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 18:43:38.402132 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980676253068-24012-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980676253068-24012-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980676253068-24012-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39725
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980676253068-24012-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39725
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:43:38.718362 13159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:43:38.718640 13159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:43:38.718757 13159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:43:38.726563 13159 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:43:38.726691 13159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:43:38.726779 13159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:43:38.726854 13159 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:43:38.726894 13159 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:43:38.737138 13159 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980676253068-24012-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980676253068-24012-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39725
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980676253068-24012-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39725
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980676253068-24012-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:43:38.738476 13159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:43:38.739730 13159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:43:38.751068 13164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:38.751190 13165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:38.751132 13167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:38.751854 13159 server_base.cc:1034] running on GCE node
I20260620 18:43:38.752558 13159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:43:38.753922 13159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:43:38.755142 13159 hybrid_clock.cc:648] HybridClock initialized: now 1781981018755077 us; error 43 us; skew 500 ppm
I20260620 18:43:38.755496 13159 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:43:38 dist-test-slave-d645 krb5kdc[13146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981018, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:43:38.761255 13159 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 18:43:38.762827 13159 webserver.cc:460] Webserver started at http://127.0.0.1:45305/ using document root <none> and password file <none>
I20260620 18:43:38.763528 13159 fs_manager.cc:359] Metadata directory not provided
I20260620 18:43:38.763644 13159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:43:38.763926 13159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:43:38.766811 13159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980676253068-24012-0/minicluster-data/master-0/data/instance:
uuid: "a074aa86d528461d8b0db2debabe4846"
format_stamp: "Formatted at 2026-06-20 18:43:38 on dist-test-slave-d645"
server_key: "9629e7f5c21c24b8953f61f64fc49239"
server_key_iv: "e23d07622b481d6151471f8bb94ad9eb"
server_key_version: "encryptionkey@0"
I20260620 18:43:38.767728 13159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980676253068-24012-0/minicluster-data/master-0/wal/instance:
uuid: "a074aa86d528461d8b0db2debabe4846"
format_stamp: "Formatted at 2026-06-20 18:43:38 on dist-test-slave-d645"
server_key: "9629e7f5c21c24b8953f61f64fc49239"
server_key_iv: "e23d07622b481d6151471f8bb94ad9eb"
server_key_version: "encryptionkey@0"
I20260620 18:43:38.772872 13159 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:43:38.776247 13174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:38.777552 13159 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:43:38.777712 13159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980676253068-24012-0/minicluster-data/master-0/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980676253068-24012-0/minicluster-data/master-0/wal
uuid: "a074aa86d528461d8b0db2debabe4846"
format_stamp: "Formatted at 2026-06-20 18:43:38 on dist-test-slave-d645"
server_key: "9629e7f5c21c24b8953f61f64fc49239"
server_key_iv: "e23d07622b481d6151471f8bb94ad9eb"
server_key_version: "encryptionkey@0"
I20260620 18:43:38.777896 13159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980676253068-24012-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980676253068-24012-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980676253068-24012-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:43:38.783469 13159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:43:38.787441 13159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:43:38.787727 13159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:43:38.804421 13159 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39725
I20260620 18:43:38.804458 13225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39725 every 8 connection(s)
I20260620 18:43:38.805811 13159 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
I20260620 18:43:38.809509 13226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:43:38.815659 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 13159
I20260620 18:43:38.815851 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980676253068-24012-0/minicluster-data/master-0/wal/instance
I20260620 18:43:38.816380 24012 external_mini_cluster.cc:1428] Setting key bc03cddfe8360e92bf154bdc65eeb813
I20260620 18:43:38.820262 13226 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a074aa86d528461d8b0db2debabe4846: Bootstrap starting.
I20260620 18:43:38.823611 13226 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a074aa86d528461d8b0db2debabe4846: Neither blocks nor log segments found. Creating new log.
I20260620 18:43:38.823714 13229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:38.818173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.21:39061 (local address 127.0.0.1:39725)
0620 18:43:38.818966 (+ 793us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:38.818980 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:38.819024 (+ 44us) server_negotiation.cc:407] Connection header received
0620 18:43:38.820663 (+ 1639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:38.820684 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:38.820986 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:38.821280 (+ 294us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:38.822236 (+ 956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:38.822241 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:38.822253 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:43:38.822267 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:38.822318 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:43:38.822343 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:38.822495 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:38.822616 (+ 121us) server_negotiation.cc:299] Negotiation successful
0620 18:43:38.822789 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":340,"threads_started":1}
I20260620 18:43:38.824651 13226 log.cc:824] T 00000000000000000000000000000000 P a074aa86d528461d8b0db2debabe4846: Log is configured to *not* fsync() on all Append() calls
I20260620 18:43:38.827205 24012 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:43:38.827584 13226 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a074aa86d528461d8b0db2debabe4846: No bootstrap required, opened a new log
I20260620 18:43:38.836546 13226 raft_consensus.cc:348] T 00000000000000000000000000000000 P a074aa86d528461d8b0db2debabe4846 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a074aa86d528461d8b0db2debabe4846" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39725 } }
I20260620 18:43:38.837127 13226 raft_consensus.cc:374] T 00000000000000000000000000000000 P a074aa86d528461d8b0db2debabe4846 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:43:38.837570 13226 raft_consensus.cc:729] T 00000000000000000000000000000000 P a074aa86d528461d8b0db2debabe4846 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a074aa86d528461d8b0db2debabe4846, State: Initialized, Role: FOLLOWER
I20260620 18:43:38.837818 13229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:38.831430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.21:60247 (local address 127.0.0.1:39725)
0620 18:43:38.831598 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:38.831604 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:38.835443 (+ 3839us) server_negotiation.cc:407] Connection header received
0620 18:43:38.835653 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:38.835657 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:38.835722 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:38.835814 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:38.836657 (+ 843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:38.836662 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:38.836666 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:43:38.836684 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:38.836723 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:43:38.836737 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:38.836919 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:38.837034 (+ 115us) server_negotiation.cc:299] Negotiation successful
0620 18:43:38.837099 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 18:43:38.838222 13226 consensus_queue.cc:260] T 00000000000000000000000000000000 P a074aa86d528461d8b0db2debabe4846 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a074aa86d528461d8b0db2debabe4846" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39725 } }
I20260620 18:43:38.838474 13226 raft_consensus.cc:388] T 00000000000000000000000000000000 P a074aa86d528461d8b0db2debabe4846 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:43:38.838599 13226 raft_consensus.cc:482] T 00000000000000000000000000000000 P a074aa86d528461d8b0db2debabe4846 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:43:38.838773 13226 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a074aa86d528461d8b0db2debabe4846 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:43:38.841171 13226 raft_consensus.cc:504] T 00000000000000000000000000000000 P a074aa86d528461d8b0db2debabe4846 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a074aa86d528461d8b0db2debabe4846" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39725 } }
I20260620 18:43:38.841673 13226 leader_election.cc:302] T 00000000000000000000000000000000 P a074aa86d528461d8b0db2debabe4846 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a074aa86d528461d8b0db2debabe4846; no voters:
I20260620 18:43:38.842315 13226 leader_election.cc:288] T 00000000000000000000000000000000 P a074aa86d528461d8b0db2debabe4846 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:43:38.842432 13236 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a074aa86d528461d8b0db2debabe4846 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:43:38.843114 13236 raft_consensus.cc:686] T 00000000000000000000000000000000 P a074aa86d528461d8b0db2debabe4846 [term 1 LEADER]: Becoming Leader. State: Replica: a074aa86d528461d8b0db2debabe4846, State: Running, Role: LEADER
I20260620 18:43:38.843660 13236 consensus_queue.cc:237] T 00000000000000000000000000000000 P a074aa86d528461d8b0db2debabe4846 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a074aa86d528461d8b0db2debabe4846" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39725 } }
I20260620 18:43:38.844641 13226 sys_catalog.cc:564] T 00000000000000000000000000000000 P a074aa86d528461d8b0db2debabe4846 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:43:38.846529 13237 sys_catalog.cc:455] T 00000000000000000000000000000000 P a074aa86d528461d8b0db2debabe4846 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a074aa86d528461d8b0db2debabe4846" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a074aa86d528461d8b0db2debabe4846" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39725 } } }
I20260620 18:43:38.846592 13238 sys_catalog.cc:455] T 00000000000000000000000000000000 P a074aa86d528461d8b0db2debabe4846 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a074aa86d528461d8b0db2debabe4846. Latest consensus state: current_term: 1 leader_uuid: "a074aa86d528461d8b0db2debabe4846" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a074aa86d528461d8b0db2debabe4846" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39725 } } }
I20260620 18:43:38.846836 13237 sys_catalog.cc:458] T 00000000000000000000000000000000 P a074aa86d528461d8b0db2debabe4846 [sys.catalog]: This master's current role is: LEADER
I20260620 18:43:38.846908 13238 sys_catalog.cc:458] T 00000000000000000000000000000000 P a074aa86d528461d8b0db2debabe4846 [sys.catalog]: This master's current role is: LEADER
I20260620 18:43:38.848857 13242 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:43:38.854118 13242 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:43:38.862747 13242 catalog_manager.cc:1347] Generated new cluster ID: 56afc4ebc73d47f7b6ff6932b6af30c8
I20260620 18:43:38.862864 13242 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:43:38.905586 13242 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:43:38.905982 13242 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:43:38.920462 13242 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a074aa86d528461d8b0db2debabe4846: Generated new TSK 0
I20260620 18:43:38.921489 13242 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:43:38.941620 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 13159
W20260620 18:43:38.942795 13221 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.21:39061
W20260620 18:43:38.943603 24012 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 13159: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:43:38.943753 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 13159
2026-06-20T18:43:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1810 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980676253068-24012-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:43:39 dist-test-slave-d645 krb5kdc[13257](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:43:39 dist-test-slave-d645 krb5kdc[13257](info): set up 2 sockets
Jun 20 18:43:39 dist-test-slave-d645 krb5kdc[13257](info): commencing operation
krb5kdc: starting...
W20260620 18:43:39.971351 24012 mini_kdc.cc:121] Time spent starting KDC: real 0.996s user 0.004s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980676253068-24012-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980676253068-24012-0/krb5kdc/test-user.keytab.
Jun 20 18:43:40 dist-test-slave-d645 krb5kdc[13257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:43:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:43:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.115.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.115.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.115.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab.
Entry for principal kudu/127.23.115.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab.
I20260620 18:43:40.226785 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980676253068-24012-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980676253068-24012-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980676253068-24012-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.115.62:39677
--webserver_interface=127.23.115.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab
--principal=kudu/127.23.115.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:37495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.115.62:39677
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:43:40.552500 13270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:43:40.552776 13270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:43:40.552910 13270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:43:40.560063 13270 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:43:40.560169 13270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:43:40.560228 13270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:43:40.560312 13270 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:43:40.560371 13270 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:43:40.570125 13270 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:37495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980676253068-24012-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980676253068-24012-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.115.62:39677
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab
--principal=kudu/127.23.115.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.115.62:39677
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.115.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980676253068-24012-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:43:40.571401 13270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:43:40.572675 13270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:43:40.583695 13275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:40.583738 13278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:40.583973 13276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:40.585017 13270 server_base.cc:1034] running on GCE node
I20260620 18:43:40.585755 13270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:43:40.587116 13270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:43:40.588344 13270 hybrid_clock.cc:648] HybridClock initialized: now 1781981020588288 us; error 41 us; skew 500 ppm
I20260620 18:43:40.588692 13270 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:43:40 dist-test-slave-d645 krb5kdc[13257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981020, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:43:40.591758 13270 init.cc:377] Logged in from keytab as kudu/127.23.115.62@KRBTEST.COM (short username kudu)
I20260620 18:43:40.593226 13270 webserver.cc:460] Webserver started at http://127.23.115.62:38801/ using document root <none> and password file <none>
I20260620 18:43:40.593842 13270 fs_manager.cc:359] Metadata directory not provided
I20260620 18:43:40.593937 13270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:43:40.594183 13270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:43:40.596993 13270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980676253068-24012-0/minicluster-data/master-0/data/instance:
uuid: "433ce8991f6a40648c7b52175b81d5fe"
format_stamp: "Formatted at 2026-06-20 18:43:40 on dist-test-slave-d645"
server_key: "27d635f96b6fc65860aab74d1c25cf31"
server_key_iv: "5a0066be357e778d6fd615d3b5a7b703"
server_key_version: "encryptionkey@0"
I20260620 18:43:40.597697 13270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980676253068-24012-0/minicluster-data/master-0/wal/instance:
uuid: "433ce8991f6a40648c7b52175b81d5fe"
format_stamp: "Formatted at 2026-06-20 18:43:40 on dist-test-slave-d645"
server_key: "27d635f96b6fc65860aab74d1c25cf31"
server_key_iv: "5a0066be357e778d6fd615d3b5a7b703"
server_key_version: "encryptionkey@0"
I20260620 18:43:40.602753 13270 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260620 18:43:40.607234 13285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:40.608537 13270 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:43:40.608712 13270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980676253068-24012-0/minicluster-data/master-0/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980676253068-24012-0/minicluster-data/master-0/wal
uuid: "433ce8991f6a40648c7b52175b81d5fe"
format_stamp: "Formatted at 2026-06-20 18:43:40 on dist-test-slave-d645"
server_key: "27d635f96b6fc65860aab74d1c25cf31"
server_key_iv: "5a0066be357e778d6fd615d3b5a7b703"
server_key_version: "encryptionkey@0"
I20260620 18:43:40.608887 13270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980676253068-24012-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980676253068-24012-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980676253068-24012-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:43:40.626111 13270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:43:40.630100 13270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:43:40.630376 13270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:43:40.647766 13270 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.62:39677
I20260620 18:43:40.647789 13336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.62:39677 every 8 connection(s)
I20260620 18:43:40.649053 13270 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
I20260620 18:43:40.650771 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 13270
I20260620 18:43:40.650965 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980676253068-24012-0/minicluster-data/master-0/wal/instance
I20260620 18:43:40.651400 24012 external_mini_cluster.cc:1428] Setting key 0dfc1fd34145ec724a809d67360fe51b
I20260620 18:43:40.652923 13337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 18:43:40 dist-test-slave-d645 krb5kdc[13257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.115.62@KRBTEST.COM
I20260620 18:43:40.666152 13337 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 433ce8991f6a40648c7b52175b81d5fe: Bootstrap starting.
I20260620 18:43:40.669400 13337 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 433ce8991f6a40648c7b52175b81d5fe: Neither blocks nor log segments found. Creating new log.
I20260620 18:43:40.670624 13337 log.cc:824] T 00000000000000000000000000000000 P 433ce8991f6a40648c7b52175b81d5fe: Log is configured to *not* fsync() on all Append() calls
I20260620 18:43:40.673421 13337 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 433ce8991f6a40648c7b52175b81d5fe: No bootstrap required, opened a new log
I20260620 18:43:40.673769 13340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:40.652986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47355 (local address 127.23.115.62:39677)
0620 18:43:40.653725 (+ 739us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:40.653737 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:40.653775 (+ 38us) server_negotiation.cc:407] Connection header received
0620 18:43:40.655369 (+ 1594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:40.655388 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:40.655768 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:40.656077 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:40.657884 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:40.659319 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:40.661051 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:40.661735 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:40.661864 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:40.666564 (+ 4700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:40.666571 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:40.666585 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:40.666619 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:40.669517 (+ 2898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:40.670448 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:40.670454 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:40.670464 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:40.670644 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:40.671207 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:40.671211 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:40.671213 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:40.671728 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:40.672025 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:40.672544 (+ 519us) server_negotiation.cc:299] Negotiation successful
0620 18:43:40.672834 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":315,"threads_started":1}
I20260620 18:43:40.677364 24012 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:43:40.681075 13337 raft_consensus.cc:348] T 00000000000000000000000000000000 P 433ce8991f6a40648c7b52175b81d5fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "433ce8991f6a40648c7b52175b81d5fe" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 39677 } }
I20260620 18:43:40.681485 13337 raft_consensus.cc:374] T 00000000000000000000000000000000 P 433ce8991f6a40648c7b52175b81d5fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:43:40.681641 13337 raft_consensus.cc:729] T 00000000000000000000000000000000 P 433ce8991f6a40648c7b52175b81d5fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 433ce8991f6a40648c7b52175b81d5fe, State: Initialized, Role: FOLLOWER
I20260620 18:43:40.682476 13337 consensus_queue.cc:260] T 00000000000000000000000000000000 P 433ce8991f6a40648c7b52175b81d5fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "433ce8991f6a40648c7b52175b81d5fe" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 39677 } }
I20260620 18:43:40.682816 13337 raft_consensus.cc:388] T 00000000000000000000000000000000 P 433ce8991f6a40648c7b52175b81d5fe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:43:40.682968 13337 raft_consensus.cc:482] T 00000000000000000000000000000000 P 433ce8991f6a40648c7b52175b81d5fe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:43:40.683141 13337 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 433ce8991f6a40648c7b52175b81d5fe [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:43:40.685931 13337 raft_consensus.cc:504] T 00000000000000000000000000000000 P 433ce8991f6a40648c7b52175b81d5fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "433ce8991f6a40648c7b52175b81d5fe" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 39677 } }
I20260620 18:43:40.686519 13337 leader_election.cc:302] T 00000000000000000000000000000000 P 433ce8991f6a40648c7b52175b81d5fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 433ce8991f6a40648c7b52175b81d5fe; no voters:
I20260620 18:43:40.687186 13337 leader_election.cc:288] T 00000000000000000000000000000000 P 433ce8991f6a40648c7b52175b81d5fe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:43:40.687280 13347 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 433ce8991f6a40648c7b52175b81d5fe [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:43:40.689471 13337 sys_catalog.cc:564] T 00000000000000000000000000000000 P 433ce8991f6a40648c7b52175b81d5fe [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:43:40.690572 13347 raft_consensus.cc:686] T 00000000000000000000000000000000 P 433ce8991f6a40648c7b52175b81d5fe [term 1 LEADER]: Becoming Leader. State: Replica: 433ce8991f6a40648c7b52175b81d5fe, State: Running, Role: LEADER
I20260620 18:43:40.691322 13347 consensus_queue.cc:237] T 00000000000000000000000000000000 P 433ce8991f6a40648c7b52175b81d5fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "433ce8991f6a40648c7b52175b81d5fe" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 39677 } }
I20260620 18:43:40.697106 13340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:40.681394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58311 (local address 127.23.115.62:39677)
0620 18:43:40.681593 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:40.681599 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:40.682027 (+ 428us) server_negotiation.cc:407] Connection header received
0620 18:43:40.682166 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:40.682169 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:40.682226 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:40.682302 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:40.684168 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:40.684951 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:40.686565 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:40.687104 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:40.687243 (+ 139us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:40.689765 (+ 2522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:40.689770 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:40.689773 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:40.689802 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:40.693892 (+ 4090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:40.695015 (+ 1123us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:40.695020 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:40.695024 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:40.695236 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:40.695750 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:40.695753 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:40.695755 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:40.696023 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:40.696236 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:40.696681 (+ 445us) server_negotiation.cc:299] Negotiation successful
0620 18:43:40.696806 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260620 18:43:40.702168 13348 sys_catalog.cc:455] T 00000000000000000000000000000000 P 433ce8991f6a40648c7b52175b81d5fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "433ce8991f6a40648c7b52175b81d5fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "433ce8991f6a40648c7b52175b81d5fe" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 39677 } } }
I20260620 18:43:40.702529 13348 sys_catalog.cc:458] T 00000000000000000000000000000000 P 433ce8991f6a40648c7b52175b81d5fe [sys.catalog]: This master's current role is: LEADER
I20260620 18:43:40.702334 13349 sys_catalog.cc:455] T 00000000000000000000000000000000 P 433ce8991f6a40648c7b52175b81d5fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 433ce8991f6a40648c7b52175b81d5fe. Latest consensus state: current_term: 1 leader_uuid: "433ce8991f6a40648c7b52175b81d5fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "433ce8991f6a40648c7b52175b81d5fe" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 39677 } } }
I20260620 18:43:40.702649 13349 sys_catalog.cc:458] T 00000000000000000000000000000000 P 433ce8991f6a40648c7b52175b81d5fe [sys.catalog]: This master's current role is: LEADER
I20260620 18:43:40.703271 13363 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:43:40.710453 13363 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:43:40.718410 13363 catalog_manager.cc:1347] Generated new cluster ID: 6a563a49763341969156cedb08e1386f
I20260620 18:43:40.718528 13363 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:43:40.744680 13363 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:43:40.745991 13363 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:43:40.758194 13363 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 433ce8991f6a40648c7b52175b81d5fe: Generated new TSK 0
I20260620 18:43:40.758987 13363 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:43:40.765439 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 13270
I20260620 18:43:40.767313 13332 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:43:40.857113 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 13270
W20260620 18:43:40.871235 13255 connection.cc:535] client connection to 127.23.115.62:39677 recv error: Network error: failed to read from TLS socket (remote: 127.23.115.62:39677): Connection reset by peer (error 104)
2026-06-20T18:43:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1914 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980676253068-24012-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:43:40 dist-test-slave-d645 krb5kdc[13370](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:43:40 dist-test-slave-d645 krb5kdc[13370](info): set up 2 sockets
Jun 20 18:43:40 dist-test-slave-d645 krb5kdc[13370](info): commencing operation
krb5kdc: starting...
W20260620 18:43:41.879374 24012 mini_kdc.cc:121] Time spent starting KDC: real 0.989s user 0.001s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980676253068-24012-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980676253068-24012-0/krb5kdc/test-user.keytab.
Jun 20 18:43:42 dist-test-slave-d645 krb5kdc[13370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981022, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:43:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:43:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.115.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.115.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.115.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab.
Entry for principal kudu/127.23.115.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab.
I20260620 18:43:42.130259 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980676253068-24012-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980676253068-24012-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980676253068-24012-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.115.62:37439
--webserver_interface=127.23.115.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab
--principal=kudu/127.23.115.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:34555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.115.62:37439
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:43:42.452046 13383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:43:42.452333 13383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:43:42.452469 13383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:43:42.459769 13383 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:43:42.459885 13383 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:43:42.459942 13383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:43:42.460026 13383 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:43:42.460095 13383 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:43:42.470175 13383 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:34555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980676253068-24012-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980676253068-24012-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.115.62:37439
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab
--principal=kudu/127.23.115.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.115.62:37439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.115.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980676253068-24012-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:43:42.471501 13383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:43:42.472764 13383 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:43:42.483912 13388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:42.483911 13389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:42.483927 13391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:42.484814 13383 server_base.cc:1034] running on GCE node
I20260620 18:43:42.485581 13383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:43:42.486995 13383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:43:42.488220 13383 hybrid_clock.cc:648] HybridClock initialized: now 1781981022488170 us; error 42 us; skew 500 ppm
I20260620 18:43:42.488561 13383 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:43:42 dist-test-slave-d645 krb5kdc[13370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981022, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:43:42.491458 13383 init.cc:377] Logged in from keytab as kudu/127.23.115.62@KRBTEST.COM (short username kudu)
I20260620 18:43:42.492826 13383 webserver.cc:460] Webserver started at http://127.23.115.62:33545/ using document root <none> and password file <none>
I20260620 18:43:42.493458 13383 fs_manager.cc:359] Metadata directory not provided
I20260620 18:43:42.493554 13383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:43:42.493806 13383 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:43:42.496627 13383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980676253068-24012-0/minicluster-data/master-0/data/instance:
uuid: "dbba9bac1cf7457ab26feb523cf4afee"
format_stamp: "Formatted at 2026-06-20 18:43:42 on dist-test-slave-d645"
server_key: "178ab0e90b1680287dc8f3e40daf96c8"
server_key_iv: "3ea546d69e02b9cb657fa2668f822d1b"
server_key_version: "encryptionkey@0"
I20260620 18:43:42.497332 13383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980676253068-24012-0/minicluster-data/master-0/wal/instance:
uuid: "dbba9bac1cf7457ab26feb523cf4afee"
format_stamp: "Formatted at 2026-06-20 18:43:42 on dist-test-slave-d645"
server_key: "178ab0e90b1680287dc8f3e40daf96c8"
server_key_iv: "3ea546d69e02b9cb657fa2668f822d1b"
server_key_version: "encryptionkey@0"
I20260620 18:43:42.502276 13383 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 18:43:42.505829 13398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:42.507464 13383 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 18:43:42.507666 13383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980676253068-24012-0/minicluster-data/master-0/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980676253068-24012-0/minicluster-data/master-0/wal
uuid: "dbba9bac1cf7457ab26feb523cf4afee"
format_stamp: "Formatted at 2026-06-20 18:43:42 on dist-test-slave-d645"
server_key: "178ab0e90b1680287dc8f3e40daf96c8"
server_key_iv: "3ea546d69e02b9cb657fa2668f822d1b"
server_key_version: "encryptionkey@0"
I20260620 18:43:42.507856 13383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980676253068-24012-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980676253068-24012-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980676253068-24012-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:43:42.524163 13383 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:43:42.528227 13383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:43:42.528478 13383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:43:42.545665 13383 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.62:37439
I20260620 18:43:42.545706 13449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.62:37439 every 8 connection(s)
I20260620 18:43:42.547022 13383 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
I20260620 18:43:42.550601 13450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:43:42.553894 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 13383
I20260620 18:43:42.554061 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980676253068-24012-0/minicluster-data/master-0/wal/instance
I20260620 18:43:42.554472 24012 external_mini_cluster.cc:1428] Setting key 3da09ac3213caa0257e2d9ce2785bce2
I20260620 18:43:42.563424 13450 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbba9bac1cf7457ab26feb523cf4afee: Bootstrap starting.
I20260620 18:43:42.567061 13450 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dbba9bac1cf7457ab26feb523cf4afee: Neither blocks nor log segments found. Creating new log.
I20260620 18:43:42.568315 13450 log.cc:824] T 00000000000000000000000000000000 P dbba9bac1cf7457ab26feb523cf4afee: Log is configured to *not* fsync() on all Append() calls
I20260620 18:43:42.568924 13453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:42.556206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49457 (local address 127.23.115.62:37439)
0620 18:43:42.556972 (+ 766us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:42.556986 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:42.557035 (+ 49us) server_negotiation.cc:407] Connection header received
0620 18:43:42.559076 (+ 2041us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:42.559104 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:42.559601 (+ 497us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:42.560025 (+ 424us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:42.560523 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:42.562462 (+ 1939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:42.563951 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:42.564924 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:42.565128 (+ 204us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:42.566248 (+ 1120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:42.566258 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:42.566275 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:43:42.566917 (+ 642us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:42.567037 (+ 120us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:43:42.567080 (+ 43us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:42.567225 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:42.567491 (+ 266us) server_negotiation.cc:299] Negotiation successful
0620 18:43:42.567657 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":325,"threads_started":1}
I20260620 18:43:42.571416 13450 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbba9bac1cf7457ab26feb523cf4afee: No bootstrap required, opened a new log
I20260620 18:43:42.572948 24012 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:43:42.581079 13450 raft_consensus.cc:348] T 00000000000000000000000000000000 P dbba9bac1cf7457ab26feb523cf4afee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbba9bac1cf7457ab26feb523cf4afee" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 37439 } }
I20260620 18:43:42.581414 13450 raft_consensus.cc:374] T 00000000000000000000000000000000 P dbba9bac1cf7457ab26feb523cf4afee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:43:42.581521 13450 raft_consensus.cc:729] T 00000000000000000000000000000000 P dbba9bac1cf7457ab26feb523cf4afee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbba9bac1cf7457ab26feb523cf4afee, State: Initialized, Role: FOLLOWER
I20260620 18:43:42.582070 13450 consensus_queue.cc:260] T 00000000000000000000000000000000 P dbba9bac1cf7457ab26feb523cf4afee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbba9bac1cf7457ab26feb523cf4afee" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 37439 } }
I20260620 18:43:42.582296 13450 raft_consensus.cc:388] T 00000000000000000000000000000000 P dbba9bac1cf7457ab26feb523cf4afee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:43:42.582429 13450 raft_consensus.cc:482] T 00000000000000000000000000000000 P dbba9bac1cf7457ab26feb523cf4afee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:43:42.582592 13450 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dbba9bac1cf7457ab26feb523cf4afee [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:43:42.584357 13453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:42.578026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47045 (local address 127.23.115.62:37439)
0620 18:43:42.578204 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:42.578209 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:42.578538 (+ 329us) server_negotiation.cc:407] Connection header received
0620 18:43:42.578714 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:42.578720 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:42.578793 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:42.578905 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:42.579391 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:42.580520 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:42.581905 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:42.582375 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:42.582493 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:42.583629 (+ 1136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:42.583634 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:42.583637 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:43:42.583654 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:42.583717 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:43:42.583735 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:42.583858 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:42.584032 (+ 174us) server_negotiation.cc:299] Negotiation successful
0620 18:43:42.584098 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 18:43:42.585134 13450 raft_consensus.cc:504] T 00000000000000000000000000000000 P dbba9bac1cf7457ab26feb523cf4afee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbba9bac1cf7457ab26feb523cf4afee" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 37439 } }
I20260620 18:43:42.585687 13450 leader_election.cc:302] T 00000000000000000000000000000000 P dbba9bac1cf7457ab26feb523cf4afee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dbba9bac1cf7457ab26feb523cf4afee; no voters:
I20260620 18:43:42.586262 13450 leader_election.cc:288] T 00000000000000000000000000000000 P dbba9bac1cf7457ab26feb523cf4afee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:43:42.586478 13460 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dbba9bac1cf7457ab26feb523cf4afee [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:43:42.587433 13460 raft_consensus.cc:686] T 00000000000000000000000000000000 P dbba9bac1cf7457ab26feb523cf4afee [term 1 LEADER]: Becoming Leader. State: Replica: dbba9bac1cf7457ab26feb523cf4afee, State: Running, Role: LEADER
I20260620 18:43:42.588063 13460 consensus_queue.cc:237] T 00000000000000000000000000000000 P dbba9bac1cf7457ab26feb523cf4afee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbba9bac1cf7457ab26feb523cf4afee" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 37439 } }
I20260620 18:43:42.588788 13450 sys_catalog.cc:564] T 00000000000000000000000000000000 P dbba9bac1cf7457ab26feb523cf4afee [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:43:42.590605 13461 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbba9bac1cf7457ab26feb523cf4afee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dbba9bac1cf7457ab26feb523cf4afee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbba9bac1cf7457ab26feb523cf4afee" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 37439 } } }
I20260620 18:43:42.590977 13461 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbba9bac1cf7457ab26feb523cf4afee [sys.catalog]: This master's current role is: LEADER
I20260620 18:43:42.592371 13462 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbba9bac1cf7457ab26feb523cf4afee [sys.catalog]: SysCatalogTable state changed. Reason: New leader dbba9bac1cf7457ab26feb523cf4afee. Latest consensus state: current_term: 1 leader_uuid: "dbba9bac1cf7457ab26feb523cf4afee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbba9bac1cf7457ab26feb523cf4afee" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 37439 } } }
I20260620 18:43:42.593221 13465 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:43:42.594587 13462 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbba9bac1cf7457ab26feb523cf4afee [sys.catalog]: This master's current role is: LEADER
I20260620 18:43:42.597823 13465 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:43:42.606734 13465 catalog_manager.cc:1347] Generated new cluster ID: 26a2765dd4674b959cd97c5c7de4f6a3
I20260620 18:43:42.606845 13465 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:43:42.627434 13465 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:43:42.629218 13465 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:43:42.653890 13465 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dbba9bac1cf7457ab26feb523cf4afee: Generated new TSK 0
I20260620 18:43:42.654752 13465 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:43:42.664736 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 13383
W20260620 18:43:42.665920 13445 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:49457
W20260620 18:43:42.666688 24012 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 13383: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:43:42.666806 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 13383
W20260620 18:43:42.680272 13368 connection.cc:535] client connection to 127.23.115.62:37439 recv error: Network error: failed to read from TLS socket (remote: 127.23.115.62:37439): Connection reset by peer (error 104)
2026-06-20T18:43:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1808 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980676253068-24012-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:43:42 dist-test-slave-d645 krb5kdc[13481](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:43:42 dist-test-slave-d645 krb5kdc[13481](info): set up 2 sockets
Jun 20 18:43:42 dist-test-slave-d645 krb5kdc[13481](info): commencing operation
krb5kdc: starting...
W20260620 18:43:43.687337 24012 mini_kdc.cc:121] Time spent starting KDC: real 0.989s user 0.006s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980676253068-24012-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980676253068-24012-0/krb5kdc/test-user.keytab.
Jun 20 18:43:43 dist-test-slave-d645 krb5kdc[13481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981023, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:43:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:43:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.115.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.115.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.115.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab.
Entry for principal kudu/127.23.115.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab.
I20260620 18:43:43.927704 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980676253068-24012-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980676253068-24012-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980676253068-24012-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.115.62:36107
--webserver_interface=127.23.115.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab
--principal=kudu/127.23.115.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:44995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.115.62:36107
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:43:44.252497 13494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:43:44.252807 13494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:43:44.252950 13494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:43:44.261198 13494 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:43:44.261337 13494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:43:44.261415 13494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:43:44.261497 13494 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:43:44.261557 13494 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:43:44.271509 13494 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:44995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980676253068-24012-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980676253068-24012-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.115.62:36107
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab
--principal=kudu/127.23.115.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.115.62:36107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.115.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980676253068-24012-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:43:44.272799 13494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:43:44.274027 13494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:43:44.285173 13499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:44.285189 13500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:44.285347 13502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:44.286067 13494 server_base.cc:1034] running on GCE node
I20260620 18:43:44.286798 13494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:43:44.288193 13494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:43:44.289412 13494 hybrid_clock.cc:648] HybridClock initialized: now 1781981024289374 us; error 31 us; skew 500 ppm
I20260620 18:43:44.289826 13494 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:43:44 dist-test-slave-d645 krb5kdc[13481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981024, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:43:44.296759 13494 init.cc:377] Logged in from keytab as kudu/127.23.115.62@KRBTEST.COM (short username kudu)
I20260620 18:43:44.299137 13494 webserver.cc:460] Webserver started at http://127.23.115.62:36649/ using document root <none> and password file <none>
I20260620 18:43:44.299938 13494 fs_manager.cc:359] Metadata directory not provided
I20260620 18:43:44.300067 13494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:43:44.300367 13494 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:43:44.303511 13494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980676253068-24012-0/minicluster-data/master-0/data/instance:
uuid: "5c180ea18d9342b9a46ab1b79494011a"
format_stamp: "Formatted at 2026-06-20 18:43:44 on dist-test-slave-d645"
server_key: "132250dcd6f21f3c0b58392dc69a182f"
server_key_iv: "e9f06abbae379ca9058c17a4d7d95c4c"
server_key_version: "encryptionkey@0"
I20260620 18:43:44.304302 13494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980676253068-24012-0/minicluster-data/master-0/wal/instance:
uuid: "5c180ea18d9342b9a46ab1b79494011a"
format_stamp: "Formatted at 2026-06-20 18:43:44 on dist-test-slave-d645"
server_key: "132250dcd6f21f3c0b58392dc69a182f"
server_key_iv: "e9f06abbae379ca9058c17a4d7d95c4c"
server_key_version: "encryptionkey@0"
I20260620 18:43:44.309605 13494 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:43:44.313208 13509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:44.314528 13494 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:43:44.314692 13494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980676253068-24012-0/minicluster-data/master-0/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980676253068-24012-0/minicluster-data/master-0/wal
uuid: "5c180ea18d9342b9a46ab1b79494011a"
format_stamp: "Formatted at 2026-06-20 18:43:44 on dist-test-slave-d645"
server_key: "132250dcd6f21f3c0b58392dc69a182f"
server_key_iv: "e9f06abbae379ca9058c17a4d7d95c4c"
server_key_version: "encryptionkey@0"
I20260620 18:43:44.314868 13494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980676253068-24012-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980676253068-24012-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980676253068-24012-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:43:44.320528 13494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:43:44.324568 13494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:43:44.324841 13494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:43:44.343910 13494 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.62:36107
I20260620 18:43:44.343947 13560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.62:36107 every 8 connection(s)
I20260620 18:43:44.345253 13494 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
I20260620 18:43:44.349402 13561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:43:44.353760 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 13494
I20260620 18:43:44.353967 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980676253068-24012-0/minicluster-data/master-0/wal/instance
I20260620 18:43:44.354585 24012 external_mini_cluster.cc:1428] Setting key 39087af6fcd8351621721307ecb03205
I20260620 18:43:44.360884 13561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c180ea18d9342b9a46ab1b79494011a: Bootstrap starting.
I20260620 18:43:44.363104 13564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:44.356432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50373 (local address 127.23.115.62:36107)
0620 18:43:44.357216 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:44.357228 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:44.357268 (+ 40us) server_negotiation.cc:407] Connection header received
0620 18:43:44.359667 (+ 2399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:44.359692 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:44.360042 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:44.360398 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:44.361480 (+ 1082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:44.361487 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:44.361534 (+ 47us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:43:44.361551 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:44.361614 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:43:44.361636 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:44.361815 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:44.361918 (+ 103us) server_negotiation.cc:299] Negotiation successful
0620 18:43:44.362108 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":373,"threads_started":1}
I20260620 18:43:44.364151 13561 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c180ea18d9342b9a46ab1b79494011a: Neither blocks nor log segments found. Creating new log.
I20260620 18:43:44.365125 13561 log.cc:824] T 00000000000000000000000000000000 P 5c180ea18d9342b9a46ab1b79494011a: Log is configured to *not* fsync() on all Append() calls
I20260620 18:43:44.366641 24012 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:43:44.368000 13561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c180ea18d9342b9a46ab1b79494011a: No bootstrap required, opened a new log
I20260620 18:43:44.373306 13564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:44.370819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57409 (local address 127.23.115.62:36107)
0620 18:43:44.371029 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:44.371036 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:44.371559 (+ 523us) server_negotiation.cc:407] Connection header received
0620 18:43:44.371678 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:44.371682 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:44.371729 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:44.371811 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:44.372676 (+ 865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:44.372681 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:44.372684 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:43:44.372698 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:44.372739 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:43:44.372754 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:44.372912 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:44.372985 (+ 73us) server_negotiation.cc:299] Negotiation successful
0620 18:43:44.373038 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 18:43:44.376322 13561 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c180ea18d9342b9a46ab1b79494011a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c180ea18d9342b9a46ab1b79494011a" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 36107 } }
I20260620 18:43:44.376667 13561 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5c180ea18d9342b9a46ab1b79494011a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:43:44.376763 13561 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c180ea18d9342b9a46ab1b79494011a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c180ea18d9342b9a46ab1b79494011a, State: Initialized, Role: FOLLOWER
I20260620 18:43:44.377300 13561 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c180ea18d9342b9a46ab1b79494011a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c180ea18d9342b9a46ab1b79494011a" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 36107 } }
I20260620 18:43:44.377519 13561 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5c180ea18d9342b9a46ab1b79494011a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:43:44.377648 13561 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c180ea18d9342b9a46ab1b79494011a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:43:44.377801 13561 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c180ea18d9342b9a46ab1b79494011a [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:43:44.380141 13561 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c180ea18d9342b9a46ab1b79494011a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c180ea18d9342b9a46ab1b79494011a" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 36107 } }
I20260620 18:43:44.380805 13561 leader_election.cc:302] T 00000000000000000000000000000000 P 5c180ea18d9342b9a46ab1b79494011a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5c180ea18d9342b9a46ab1b79494011a; no voters:
I20260620 18:43:44.381394 13561 leader_election.cc:288] T 00000000000000000000000000000000 P 5c180ea18d9342b9a46ab1b79494011a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:43:44.381537 13571 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c180ea18d9342b9a46ab1b79494011a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:43:44.383359 13561 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c180ea18d9342b9a46ab1b79494011a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:43:44.388172 13571 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5c180ea18d9342b9a46ab1b79494011a [term 1 LEADER]: Becoming Leader. State: Replica: 5c180ea18d9342b9a46ab1b79494011a, State: Running, Role: LEADER
I20260620 18:43:44.388641 13571 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c180ea18d9342b9a46ab1b79494011a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c180ea18d9342b9a46ab1b79494011a" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 36107 } }
I20260620 18:43:44.391419 13573 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c180ea18d9342b9a46ab1b79494011a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c180ea18d9342b9a46ab1b79494011a. Latest consensus state: current_term: 1 leader_uuid: "5c180ea18d9342b9a46ab1b79494011a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c180ea18d9342b9a46ab1b79494011a" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 36107 } } }
I20260620 18:43:44.391861 13573 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c180ea18d9342b9a46ab1b79494011a [sys.catalog]: This master's current role is: LEADER
I20260620 18:43:44.391448 13572 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c180ea18d9342b9a46ab1b79494011a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5c180ea18d9342b9a46ab1b79494011a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c180ea18d9342b9a46ab1b79494011a" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 36107 } } }
I20260620 18:43:44.392305 13572 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c180ea18d9342b9a46ab1b79494011a [sys.catalog]: This master's current role is: LEADER
W20260620 18:43:44.396457 13585 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5c180ea18d9342b9a46ab1b79494011a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 18:43:44.396565 13585 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 18:43:44.396679 13586 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:43:44.398774 13586 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:43:44.407331 13586 catalog_manager.cc:1347] Generated new cluster ID: a25c7e1d4dea4cd5a9086b69b73f1775
I20260620 18:43:44.407434 13586 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:43:44.437455 13586 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:43:44.437870 13586 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:43:44.449432 13586 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5c180ea18d9342b9a46ab1b79494011a: Generated new TSK 0
I20260620 18:43:44.450292 13586 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:43:44.460587 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 13494
W20260620 18:43:44.461736 13556 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:50373
W20260620 18:43:44.462376 24012 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 13494: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:43:44.462499 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 13494
2026-06-20T18:43:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1796 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980676253068-24012-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:43:44 dist-test-slave-d645 krb5kdc[13592](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:43:44 dist-test-slave-d645 krb5kdc[13592](info): set up 2 sockets
Jun 20 18:43:44 dist-test-slave-d645 krb5kdc[13592](info): commencing operation
krb5kdc: starting...
W20260620 18:43:45.487382 24012 mini_kdc.cc:121] Time spent starting KDC: real 0.992s user 0.008s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980676253068-24012-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980676253068-24012-0/krb5kdc/test-user.keytab.
Jun 20 18:43:45 dist-test-slave-d645 krb5kdc[13592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981025, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:43:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:43:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.115.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.115.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.115.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab.
Entry for principal kudu/127.23.115.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab.
I20260620 18:43:45.730084 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980676253068-24012-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980676253068-24012-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980676253068-24012-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.115.62:32819
--webserver_interface=127.23.115.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab
--principal=kudu/127.23.115.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:34021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.115.62:32819
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:43:46.075194 13605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:43:46.075505 13605 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:43:46.075626 13605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:43:46.082902 13605 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:43:46.083042 13605 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:43:46.083117 13605 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:43:46.083199 13605 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:43:46.083271 13605 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:43:46.093333 13605 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:34021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980676253068-24012-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980676253068-24012-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.115.62:32819
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab
--principal=kudu/127.23.115.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.115.62:32819
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.115.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980676253068-24012-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:43:46.094688 13605 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:43:46.095992 13605 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:43:46.107650 13613 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:46.107784 13610 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:46.107766 13611 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:46.108882 13605 server_base.cc:1034] running on GCE node
I20260620 18:43:46.109823 13605 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:43:46.111320 13605 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:43:46.112555 13605 hybrid_clock.cc:648] HybridClock initialized: now 1781981026112501 us; error 36 us; skew 500 ppm
I20260620 18:43:46.112933 13605 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:43:46 dist-test-slave-d645 krb5kdc[13592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981026, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:43:46.115993 13605 init.cc:377] Logged in from keytab as kudu/127.23.115.62@KRBTEST.COM (short username kudu)
I20260620 18:43:46.117511 13605 webserver.cc:460] Webserver started at http://127.23.115.62:46159/ using document root <none> and password file <none>
I20260620 18:43:46.118181 13605 fs_manager.cc:359] Metadata directory not provided
I20260620 18:43:46.118297 13605 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:43:46.118571 13605 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:43:46.121819 13605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980676253068-24012-0/minicluster-data/master-0/data/instance:
uuid: "0e9092f20eed4398844e8b94df7bdc74"
format_stamp: "Formatted at 2026-06-20 18:43:46 on dist-test-slave-d645"
server_key: "3fbd8a4233318e55ea6c5bbd7f6dcaaa"
server_key_iv: "a7c257ed880fcd9b1fbe6bd80cb67dc0"
server_key_version: "encryptionkey@0"
I20260620 18:43:46.122639 13605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980676253068-24012-0/minicluster-data/master-0/wal/instance:
uuid: "0e9092f20eed4398844e8b94df7bdc74"
format_stamp: "Formatted at 2026-06-20 18:43:46 on dist-test-slave-d645"
server_key: "3fbd8a4233318e55ea6c5bbd7f6dcaaa"
server_key_iv: "a7c257ed880fcd9b1fbe6bd80cb67dc0"
server_key_version: "encryptionkey@0"
I20260620 18:43:46.128144 13605 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 18:43:46.131811 13620 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:46.133538 13605 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:43:46.133726 13605 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980676253068-24012-0/minicluster-data/master-0/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980676253068-24012-0/minicluster-data/master-0/wal
uuid: "0e9092f20eed4398844e8b94df7bdc74"
format_stamp: "Formatted at 2026-06-20 18:43:46 on dist-test-slave-d645"
server_key: "3fbd8a4233318e55ea6c5bbd7f6dcaaa"
server_key_iv: "a7c257ed880fcd9b1fbe6bd80cb67dc0"
server_key_version: "encryptionkey@0"
I20260620 18:43:46.133930 13605 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980676253068-24012-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980676253068-24012-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980676253068-24012-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:43:46.155966 13605 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:43:46.160288 13605 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:43:46.160543 13605 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:43:46.179629 13605 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.62:32819
I20260620 18:43:46.179692 13671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.62:32819 every 8 connection(s)
I20260620 18:43:46.180954 13605 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
I20260620 18:43:46.184514 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 13605
I20260620 18:43:46.184701 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980676253068-24012-0/minicluster-data/master-0/wal/instance
I20260620 18:43:46.184875 13672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:43:46.185245 24012 external_mini_cluster.cc:1428] Setting key 1597a068191ba47fc04671975547e080
I20260620 18:43:46.199407 13672 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e9092f20eed4398844e8b94df7bdc74: Bootstrap starting.
Jun 20 18:43:46 dist-test-slave-d645 krb5kdc[13592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981025, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.115.62@KRBTEST.COM
I20260620 18:43:46.203299 13672 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0e9092f20eed4398844e8b94df7bdc74: Neither blocks nor log segments found. Creating new log.
I20260620 18:43:46.204478 13672 log.cc:824] T 00000000000000000000000000000000 P 0e9092f20eed4398844e8b94df7bdc74: Log is configured to *not* fsync() on all Append() calls
I20260620 18:43:46.207840 13672 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e9092f20eed4398844e8b94df7bdc74: No bootstrap required, opened a new log
I20260620 18:43:46.209168 13675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:46.187045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48971 (local address 127.23.115.62:32819)
0620 18:43:46.187831 (+ 786us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:46.187842 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:46.187883 (+ 41us) server_negotiation.cc:407] Connection header received
0620 18:43:46.189569 (+ 1686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:46.189590 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:46.189970 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:46.190298 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:46.192017 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:46.193354 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:46.195120 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:46.195911 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:46.196051 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:46.201662 (+ 5611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:46.201671 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:46.201691 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:46.201734 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:46.204717 (+ 2983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:46.205568 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:46.205577 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:46.205590 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:46.205795 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:46.206217 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:46.206220 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:46.206222 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:46.206755 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:46.207018 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:46.207752 (+ 734us) server_negotiation.cc:299] Negotiation successful
0620 18:43:46.208174 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":583,"thread_start_us":345,"threads_started":1}
I20260620 18:43:46.212759 24012 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:43:46.216996 13672 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0e9092f20eed4398844e8b94df7bdc74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e9092f20eed4398844e8b94df7bdc74" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 32819 } }
I20260620 18:43:46.217469 13672 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0e9092f20eed4398844e8b94df7bdc74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:43:46.217581 13672 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0e9092f20eed4398844e8b94df7bdc74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e9092f20eed4398844e8b94df7bdc74, State: Initialized, Role: FOLLOWER
I20260620 18:43:46.218209 13672 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e9092f20eed4398844e8b94df7bdc74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e9092f20eed4398844e8b94df7bdc74" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 32819 } }
I20260620 18:43:46.218432 13672 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0e9092f20eed4398844e8b94df7bdc74 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:43:46.218568 13672 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0e9092f20eed4398844e8b94df7bdc74 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:43:46.218743 13672 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0e9092f20eed4398844e8b94df7bdc74 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:43:46.220988 13672 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0e9092f20eed4398844e8b94df7bdc74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e9092f20eed4398844e8b94df7bdc74" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 32819 } }
I20260620 18:43:46.221698 13672 leader_election.cc:302] T 00000000000000000000000000000000 P 0e9092f20eed4398844e8b94df7bdc74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0e9092f20eed4398844e8b94df7bdc74; no voters:
I20260620 18:43:46.222319 13672 leader_election.cc:288] T 00000000000000000000000000000000 P 0e9092f20eed4398844e8b94df7bdc74 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:43:46.222424 13682 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0e9092f20eed4398844e8b94df7bdc74 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:43:46.225126 13672 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0e9092f20eed4398844e8b94df7bdc74 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:43:46.229344 13682 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0e9092f20eed4398844e8b94df7bdc74 [term 1 LEADER]: Becoming Leader. State: Replica: 0e9092f20eed4398844e8b94df7bdc74, State: Running, Role: LEADER
I20260620 18:43:46.230098 13682 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0e9092f20eed4398844e8b94df7bdc74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e9092f20eed4398844e8b94df7bdc74" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 32819 } }
I20260620 18:43:46.233127 13683 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e9092f20eed4398844e8b94df7bdc74 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0e9092f20eed4398844e8b94df7bdc74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e9092f20eed4398844e8b94df7bdc74" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 32819 } } }
I20260620 18:43:46.233541 13683 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e9092f20eed4398844e8b94df7bdc74 [sys.catalog]: This master's current role is: LEADER
I20260620 18:43:46.233549 13675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:46.217907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60133 (local address 127.23.115.62:32819)
0620 18:43:46.218072 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:46.218078 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:46.218563 (+ 485us) server_negotiation.cc:407] Connection header received
0620 18:43:46.218786 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:46.218791 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:46.218871 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:46.218985 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:46.220342 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:46.221116 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:46.223162 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:46.223888 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:46.223970 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:46.227570 (+ 3600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:46.227577 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:46.227582 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:46.227621 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:46.230425 (+ 2804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:46.231136 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:46.231142 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:46.231145 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:46.231266 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:46.231627 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:46.231631 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:46.231633 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:46.232011 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:46.232291 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:46.232916 (+ 625us) server_negotiation.cc:299] Negotiation successful
0620 18:43:46.233161 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 18:43:46.234355 13690 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:43:46.234437 13684 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e9092f20eed4398844e8b94df7bdc74 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0e9092f20eed4398844e8b94df7bdc74. Latest consensus state: current_term: 1 leader_uuid: "0e9092f20eed4398844e8b94df7bdc74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e9092f20eed4398844e8b94df7bdc74" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 32819 } } }
I20260620 18:43:46.234758 13684 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e9092f20eed4398844e8b94df7bdc74 [sys.catalog]: This master's current role is: LEADER
I20260620 18:43:46.239979 13690 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:43:46.250166 13690 catalog_manager.cc:1347] Generated new cluster ID: 41ce9517c4b748e9bff24736555813a1
I20260620 18:43:46.250295 13690 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:43:46.263751 13690 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:43:46.265709 13690 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:43:46.280968 13690 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0e9092f20eed4398844e8b94df7bdc74: Generated new TSK 0
I20260620 18:43:46.282119 13690 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:43:46.290779 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 13605
I20260620 18:43:46.292042 13667 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:43:46.381561 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 13605
2026-06-20T18:43:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1920 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980676253068-24012-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:43:46 dist-test-slave-d645 krb5kdc[13704](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:43:46 dist-test-slave-d645 krb5kdc[13704](info): set up 2 sockets
Jun 20 18:43:46 dist-test-slave-d645 krb5kdc[13704](info): commencing operation
krb5kdc: starting...
W20260620 18:43:47.439541 24012 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.003s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980676253068-24012-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980676253068-24012-0/krb5kdc/test-user.keytab.
Jun 20 18:43:47 dist-test-slave-d645 krb5kdc[13704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981027, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:43:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:43:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.115.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.115.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.115.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab.
Entry for principal kudu/127.23.115.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab.
I20260620 18:43:47.691289 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980676253068-24012-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980676253068-24012-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980676253068-24012-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.115.62:34267
--webserver_interface=127.23.115.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab
--principal=kudu/127.23.115.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:39473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.115.62:34267
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:43:48.031167 13717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:43:48.031560 13717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:43:48.031752 13717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:43:48.040961 13717 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:43:48.041093 13717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:43:48.041165 13717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:43:48.041250 13717 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:43:48.041306 13717 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:43:48.052623 13717 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:39473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980676253068-24012-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980676253068-24012-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.115.62:34267
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab
--principal=kudu/127.23.115.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.115.62:34267
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.115.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980676253068-24012-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:43:48.053948 13717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:43:48.055284 13717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:43:48.067977 13722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:48.067946 13723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:48.067950 13725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:48.069043 13717 server_base.cc:1034] running on GCE node
I20260620 18:43:48.069902 13717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:43:48.071616 13717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:43:48.073048 13717 hybrid_clock.cc:648] HybridClock initialized: now 1781981028072939 us; error 68 us; skew 500 ppm
I20260620 18:43:48.073683 13717 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:43:48 dist-test-slave-d645 krb5kdc[13704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981028, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:43:48.077459 13717 init.cc:377] Logged in from keytab as kudu/127.23.115.62@KRBTEST.COM (short username kudu)
I20260620 18:43:48.079398 13717 webserver.cc:460] Webserver started at http://127.23.115.62:36641/ using document root <none> and password file <none>
I20260620 18:43:48.080099 13717 fs_manager.cc:359] Metadata directory not provided
I20260620 18:43:48.080205 13717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:43:48.080672 13717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:43:48.084205 13717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980676253068-24012-0/minicluster-data/master-0/data/instance:
uuid: "690ec79b57364922b9aa587750d13fd9"
format_stamp: "Formatted at 2026-06-20 18:43:48 on dist-test-slave-d645"
server_key: "7185918cc1a52718877374cb345d6d34"
server_key_iv: "6e5e0de050b9b49ba3d0dc03d5e1ae6e"
server_key_version: "encryptionkey@0"
I20260620 18:43:48.084995 13717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980676253068-24012-0/minicluster-data/master-0/wal/instance:
uuid: "690ec79b57364922b9aa587750d13fd9"
format_stamp: "Formatted at 2026-06-20 18:43:48 on dist-test-slave-d645"
server_key: "7185918cc1a52718877374cb345d6d34"
server_key_iv: "6e5e0de050b9b49ba3d0dc03d5e1ae6e"
server_key_version: "encryptionkey@0"
I20260620 18:43:48.090860 13717 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 18:43:48.095736 13732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:48.097427 13717 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260620 18:43:48.097617 13717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980676253068-24012-0/minicluster-data/master-0/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980676253068-24012-0/minicluster-data/master-0/wal
uuid: "690ec79b57364922b9aa587750d13fd9"
format_stamp: "Formatted at 2026-06-20 18:43:48 on dist-test-slave-d645"
server_key: "7185918cc1a52718877374cb345d6d34"
server_key_iv: "6e5e0de050b9b49ba3d0dc03d5e1ae6e"
server_key_version: "encryptionkey@0"
I20260620 18:43:48.097824 13717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980676253068-24012-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980676253068-24012-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980676253068-24012-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:43:48.120316 13717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:43:48.125193 13717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:43:48.125665 13717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:43:48.148742 13783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.62:34267 every 8 connection(s)
I20260620 18:43:48.148758 13717 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.62:34267
I20260620 18:43:48.150488 13717 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
I20260620 18:43:48.154192 13784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:43:48.157218 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 13717
I20260620 18:43:48.157384 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980676253068-24012-0/minicluster-data/master-0/wal/instance
I20260620 18:43:48.157814 24012 external_mini_cluster.cc:1428] Setting key 5bafbba6eb8f0d32ad595ee11e77471e
I20260620 18:43:48.171211 13784 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 690ec79b57364922b9aa587750d13fd9: Bootstrap starting.
I20260620 18:43:48.174062 13787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:48.160055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52221 (local address 127.23.115.62:34267)
0620 18:43:48.160934 (+ 879us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:48.160951 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:48.161007 (+ 56us) server_negotiation.cc:407] Connection header received
0620 18:43:48.162835 (+ 1828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:48.162869 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:48.163530 (+ 661us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:48.164028 (+ 498us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:48.164686 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:48.166603 (+ 1917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:48.168123 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:48.169258 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:48.169594 (+ 336us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:48.170969 (+ 1375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:48.170983 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:48.171006 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:43:48.171788 (+ 782us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:48.171914 (+ 126us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:43:48.171953 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:48.172096 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:48.172326 (+ 230us) server_negotiation.cc:299] Negotiation successful
0620 18:43:48.172626 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":350,"threads_started":1}
I20260620 18:43:48.175758 13784 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 690ec79b57364922b9aa587750d13fd9: Neither blocks nor log segments found. Creating new log.
I20260620 18:43:48.177016 13784 log.cc:824] T 00000000000000000000000000000000 P 690ec79b57364922b9aa587750d13fd9: Log is configured to *not* fsync() on all Append() calls
I20260620 18:43:48.179172 24012 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:43:48.180362 13784 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 690ec79b57364922b9aa587750d13fd9: No bootstrap required, opened a new log
I20260620 18:43:48.189127 13784 raft_consensus.cc:348] T 00000000000000000000000000000000 P 690ec79b57364922b9aa587750d13fd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "690ec79b57364922b9aa587750d13fd9" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 34267 } }
I20260620 18:43:48.189564 13784 raft_consensus.cc:374] T 00000000000000000000000000000000 P 690ec79b57364922b9aa587750d13fd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:43:48.189755 13784 raft_consensus.cc:729] T 00000000000000000000000000000000 P 690ec79b57364922b9aa587750d13fd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 690ec79b57364922b9aa587750d13fd9, State: Initialized, Role: FOLLOWER
I20260620 18:43:48.189920 13787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:48.183599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54549 (local address 127.23.115.62:34267)
0620 18:43:48.183770 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:48.183776 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:48.184238 (+ 462us) server_negotiation.cc:407] Connection header received
0620 18:43:48.184681 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:48.184685 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:48.184756 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:48.184850 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:48.185367 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:48.186121 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:48.187455 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:48.187940 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:48.188054 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:48.189157 (+ 1103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:48.189161 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:48.189165 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:43:48.189183 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:48.189224 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:43:48.189240 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:48.189323 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:48.189451 (+ 128us) server_negotiation.cc:299] Negotiation successful
0620 18:43:48.189545 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 18:43:48.190409 13784 consensus_queue.cc:260] T 00000000000000000000000000000000 P 690ec79b57364922b9aa587750d13fd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "690ec79b57364922b9aa587750d13fd9" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 34267 } }
I20260620 18:43:48.190647 13784 raft_consensus.cc:388] T 00000000000000000000000000000000 P 690ec79b57364922b9aa587750d13fd9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:43:48.190794 13784 raft_consensus.cc:482] T 00000000000000000000000000000000 P 690ec79b57364922b9aa587750d13fd9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:43:48.190979 13784 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 690ec79b57364922b9aa587750d13fd9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:43:48.193329 13784 raft_consensus.cc:504] T 00000000000000000000000000000000 P 690ec79b57364922b9aa587750d13fd9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "690ec79b57364922b9aa587750d13fd9" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 34267 } }
I20260620 18:43:48.193894 13784 leader_election.cc:302] T 00000000000000000000000000000000 P 690ec79b57364922b9aa587750d13fd9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 690ec79b57364922b9aa587750d13fd9; no voters:
I20260620 18:43:48.194478 13784 leader_election.cc:288] T 00000000000000000000000000000000 P 690ec79b57364922b9aa587750d13fd9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:43:48.194859 13794 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 690ec79b57364922b9aa587750d13fd9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:43:48.195730 13794 raft_consensus.cc:686] T 00000000000000000000000000000000 P 690ec79b57364922b9aa587750d13fd9 [term 1 LEADER]: Becoming Leader. State: Replica: 690ec79b57364922b9aa587750d13fd9, State: Running, Role: LEADER
I20260620 18:43:48.196517 13794 consensus_queue.cc:237] T 00000000000000000000000000000000 P 690ec79b57364922b9aa587750d13fd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "690ec79b57364922b9aa587750d13fd9" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 34267 } }
I20260620 18:43:48.197167 13784 sys_catalog.cc:564] T 00000000000000000000000000000000 P 690ec79b57364922b9aa587750d13fd9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:43:48.199950 13795 sys_catalog.cc:455] T 00000000000000000000000000000000 P 690ec79b57364922b9aa587750d13fd9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "690ec79b57364922b9aa587750d13fd9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "690ec79b57364922b9aa587750d13fd9" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 34267 } } }
I20260620 18:43:48.200412 13795 sys_catalog.cc:458] T 00000000000000000000000000000000 P 690ec79b57364922b9aa587750d13fd9 [sys.catalog]: This master's current role is: LEADER
I20260620 18:43:48.200968 13796 sys_catalog.cc:455] T 00000000000000000000000000000000 P 690ec79b57364922b9aa587750d13fd9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 690ec79b57364922b9aa587750d13fd9. Latest consensus state: current_term: 1 leader_uuid: "690ec79b57364922b9aa587750d13fd9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "690ec79b57364922b9aa587750d13fd9" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 34267 } } }
I20260620 18:43:48.201339 13796 sys_catalog.cc:458] T 00000000000000000000000000000000 P 690ec79b57364922b9aa587750d13fd9 [sys.catalog]: This master's current role is: LEADER
I20260620 18:43:48.203516 13801 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:43:48.208990 13801 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:43:48.217767 13801 catalog_manager.cc:1347] Generated new cluster ID: 10159d55c8934b758c2e33baa753be28
I20260620 18:43:48.217890 13801 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:43:48.233412 13801 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:43:48.235181 13801 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:43:48.247313 13801 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 690ec79b57364922b9aa587750d13fd9: Generated new TSK 0
I20260620 18:43:48.248340 13801 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:43:48.264945 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 13717
W20260620 18:43:48.266069 13779 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:52221
W20260620 18:43:48.266763 24012 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 13717: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:43:48.266887 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 13717
2026-06-20T18:43:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1886 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980676253068-24012-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:43:48 dist-test-slave-d645 krb5kdc[13815](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:43:48 dist-test-slave-d645 krb5kdc[13815](info): set up 2 sockets
Jun 20 18:43:48 dist-test-slave-d645 krb5kdc[13815](info): commencing operation
krb5kdc: starting...
W20260620 18:43:49.303450 24012 mini_kdc.cc:121] Time spent starting KDC: real 1.000s user 0.007s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980676253068-24012-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980676253068-24012-0/krb5kdc/test-user.keytab.
Jun 20 18:43:49 dist-test-slave-d645 krb5kdc[13815](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981029, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:43:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:43:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.115.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.115.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.115.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab.
Entry for principal kudu/127.23.115.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab.
I20260620 18:43:49.551661 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980676253068-24012-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980676253068-24012-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980676253068-24012-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.115.62:32891
--webserver_interface=127.23.115.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab
--principal=kudu/127.23.115.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:46497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.115.62:32891
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:43:49.873744 13828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:43:49.874012 13828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:43:49.874132 13828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:43:49.881296 13828 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:43:49.881400 13828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:43:49.881472 13828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:43:49.881551 13828 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:43:49.881608 13828 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:43:49.891288 13828 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:46497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980676253068-24012-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980676253068-24012-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.115.62:32891
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab
--principal=kudu/127.23.115.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.115.62:32891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.115.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980676253068-24012-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:43:49.892535 13828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:43:49.893771 13828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:43:49.905583 13834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:49.905606 13833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:49.906068 13836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:49.906536 13828 server_base.cc:1034] running on GCE node
I20260620 18:43:49.907375 13828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:43:49.908854 13828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:43:49.910077 13828 hybrid_clock.cc:648] HybridClock initialized: now 1781981029910032 us; error 35 us; skew 500 ppm
I20260620 18:43:49.910419 13828 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:43:49 dist-test-slave-d645 krb5kdc[13815](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981029, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:43:49.918699 13828 init.cc:377] Logged in from keytab as kudu/127.23.115.62@KRBTEST.COM (short username kudu)
I20260620 18:43:49.920344 13828 webserver.cc:460] Webserver started at http://127.23.115.62:37649/ using document root <none> and password file <none>
I20260620 18:43:49.921051 13828 fs_manager.cc:359] Metadata directory not provided
I20260620 18:43:49.921169 13828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:43:49.921473 13828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:43:49.924589 13828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980676253068-24012-0/minicluster-data/master-0/data/instance:
uuid: "3adfc7819ca143658156020c52bbeced"
format_stamp: "Formatted at 2026-06-20 18:43:49 on dist-test-slave-d645"
server_key: "43510229571aee0b487eb5062a5c16fc"
server_key_iv: "6b504e68dcfab7d57a9d155580ceed65"
server_key_version: "encryptionkey@0"
I20260620 18:43:49.925354 13828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980676253068-24012-0/minicluster-data/master-0/wal/instance:
uuid: "3adfc7819ca143658156020c52bbeced"
format_stamp: "Formatted at 2026-06-20 18:43:49 on dist-test-slave-d645"
server_key: "43510229571aee0b487eb5062a5c16fc"
server_key_iv: "6b504e68dcfab7d57a9d155580ceed65"
server_key_version: "encryptionkey@0"
I20260620 18:43:49.930608 13828 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260620 18:43:49.934398 13843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:49.935752 13828 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 18:43:49.935956 13828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980676253068-24012-0/minicluster-data/master-0/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980676253068-24012-0/minicluster-data/master-0/wal
uuid: "3adfc7819ca143658156020c52bbeced"
format_stamp: "Formatted at 2026-06-20 18:43:49 on dist-test-slave-d645"
server_key: "43510229571aee0b487eb5062a5c16fc"
server_key_iv: "6b504e68dcfab7d57a9d155580ceed65"
server_key_version: "encryptionkey@0"
I20260620 18:43:49.936139 13828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980676253068-24012-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980676253068-24012-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980676253068-24012-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:43:49.941674 13828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:43:49.945677 13828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:43:49.945950 13828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:43:49.964076 13828 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.62:32891
I20260620 18:43:49.964099 13894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.62:32891 every 8 connection(s)
I20260620 18:43:49.965389 13828 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
I20260620 18:43:49.969002 13895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:43:49.974226 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 13828
I20260620 18:43:49.974401 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980676253068-24012-0/minicluster-data/master-0/wal/instance
I20260620 18:43:49.974826 24012 external_mini_cluster.cc:1428] Setting key 697b28037d30c42162549f2c00763cd6
I20260620 18:43:49.979470 13895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3adfc7819ca143658156020c52bbeced: Bootstrap starting.
I20260620 18:43:49.982465 13898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:49.976649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33351 (local address 127.23.115.62:32891)
0620 18:43:49.977537 (+ 888us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:49.977552 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:49.977592 (+ 40us) server_negotiation.cc:407] Connection header received
0620 18:43:49.979298 (+ 1706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:49.979322 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:49.979679 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:49.979996 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:49.981022 (+ 1026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:49.981029 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:49.981041 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:43:49.981058 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:49.981118 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:43:49.981138 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:49.981317 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:49.981416 (+ 99us) server_negotiation.cc:299] Negotiation successful
0620 18:43:49.981592 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":647,"thread_start_us":386,"threads_started":1}
I20260620 18:43:49.982959 13895 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3adfc7819ca143658156020c52bbeced: Neither blocks nor log segments found. Creating new log.
I20260620 18:43:49.983990 13895 log.cc:824] T 00000000000000000000000000000000 P 3adfc7819ca143658156020c52bbeced: Log is configured to *not* fsync() on all Append() calls
I20260620 18:43:49.985984 24012 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:43:49.986807 13895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3adfc7819ca143658156020c52bbeced: No bootstrap required, opened a new log
I20260620 18:43:49.992506 13898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:49.989986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60673 (local address 127.23.115.62:32891)
0620 18:43:49.990174 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:49.990181 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:49.990606 (+ 425us) server_negotiation.cc:407] Connection header received
0620 18:43:49.990801 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:49.990808 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:49.990860 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:49.991012 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:49.991888 (+ 876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:49.991893 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:49.991897 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:43:49.991913 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:49.991952 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:43:49.991967 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:49.992072 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:49.992189 (+ 117us) server_negotiation.cc:299] Negotiation successful
0620 18:43:49.992237 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 18:43:49.994822 13895 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3adfc7819ca143658156020c52bbeced [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3adfc7819ca143658156020c52bbeced" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 32891 } }
I20260620 18:43:49.995209 13895 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3adfc7819ca143658156020c52bbeced [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:43:49.995299 13895 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3adfc7819ca143658156020c52bbeced [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3adfc7819ca143658156020c52bbeced, State: Initialized, Role: FOLLOWER
I20260620 18:43:49.995836 13895 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3adfc7819ca143658156020c52bbeced [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3adfc7819ca143658156020c52bbeced" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 32891 } }
I20260620 18:43:49.996059 13895 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3adfc7819ca143658156020c52bbeced [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:43:49.996183 13895 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3adfc7819ca143658156020c52bbeced [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:43:49.996340 13895 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3adfc7819ca143658156020c52bbeced [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:43:49.998497 13895 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3adfc7819ca143658156020c52bbeced [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3adfc7819ca143658156020c52bbeced" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 32891 } }
I20260620 18:43:49.999044 13895 leader_election.cc:302] T 00000000000000000000000000000000 P 3adfc7819ca143658156020c52bbeced [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3adfc7819ca143658156020c52bbeced; no voters:
I20260620 18:43:49.999708 13895 leader_election.cc:288] T 00000000000000000000000000000000 P 3adfc7819ca143658156020c52bbeced [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:43:49.999836 13905 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3adfc7819ca143658156020c52bbeced [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:43:50.000437 13905 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3adfc7819ca143658156020c52bbeced [term 1 LEADER]: Becoming Leader. State: Replica: 3adfc7819ca143658156020c52bbeced, State: Running, Role: LEADER
I20260620 18:43:50.000973 13905 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3adfc7819ca143658156020c52bbeced [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3adfc7819ca143658156020c52bbeced" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 32891 } }
I20260620 18:43:50.001616 13895 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3adfc7819ca143658156020c52bbeced [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:43:50.003896 13907 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3adfc7819ca143658156020c52bbeced [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3adfc7819ca143658156020c52bbeced. Latest consensus state: current_term: 1 leader_uuid: "3adfc7819ca143658156020c52bbeced" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3adfc7819ca143658156020c52bbeced" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 32891 } } }
I20260620 18:43:50.003870 13906 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3adfc7819ca143658156020c52bbeced [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3adfc7819ca143658156020c52bbeced" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3adfc7819ca143658156020c52bbeced" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 32891 } } }
I20260620 18:43:50.004328 13907 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3adfc7819ca143658156020c52bbeced [sys.catalog]: This master's current role is: LEADER
I20260620 18:43:50.004323 13906 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3adfc7819ca143658156020c52bbeced [sys.catalog]: This master's current role is: LEADER
I20260620 18:43:50.006696 13913 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:43:50.011631 13913 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:43:50.022617 13913 catalog_manager.cc:1347] Generated new cluster ID: 54a40192fc954d54bf75aad2959b7140
I20260620 18:43:50.022825 13913 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:43:50.054764 13913 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:43:50.055368 13913 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:43:50.069852 13913 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3adfc7819ca143658156020c52bbeced: Generated new TSK 0
I20260620 18:43:50.070655 13913 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:43:50.087911 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 13828
W20260620 18:43:50.089087 13890 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:33351
W20260620 18:43:50.089746 24012 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 13828: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:43:50.089865 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 13828
2026-06-20T18:43:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1821 ms)
[----------] 18 tests from AuthTokenIssuingTest (33033 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980676253068-24012-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:43:50 dist-test-slave-d645 krb5kdc[13926](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:43:50 dist-test-slave-d645 krb5kdc[13926](info): set up 2 sockets
Jun 20 18:43:50 dist-test-slave-d645 krb5kdc[13926](info): commencing operation
krb5kdc: starting...
W20260620 18:43:51.119827 24012 mini_kdc.cc:121] Time spent starting KDC: real 0.995s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980676253068-24012-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980676253068-24012-0/krb5kdc/test-user.keytab.
Jun 20 18:43:51 dist-test-slave-d645 krb5kdc[13926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:43:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:43:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.115.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.115.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.115.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab.
Entry for principal kudu/127.23.115.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab.
I20260620 18:43:51.375074 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980676253068-24012-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980676253068-24012-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980676253068-24012-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.115.62:38193
--webserver_interface=127.23.115.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab
--principal=kudu/127.23.115.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:37753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.115.62:38193,127.23.115.61:38393,127.23.115.60:44645
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:43:51.694074 13939 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 18:43:51.694401 13939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:43:51.694475 13939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:43:51.694556 13939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:43:51.701943 13939 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:43:51.702044 13939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:43:51.702100 13939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:43:51.702148 13939 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:43:51.702208 13939 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:43:51.712858 13939 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:37753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980676253068-24012-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980676253068-24012-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.115.62:38193,127.23.115.61:38393,127.23.115.60:44645
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab
--principal=kudu/127.23.115.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.115.62:38193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.115.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980676253068-24012-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:43:51.714141 13939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:43:51.715456 13939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:43:51.727186 13944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:51.727250 13945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:51.727365 13947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:51.728194 13939 server_base.cc:1034] running on GCE node
I20260620 18:43:51.728973 13939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:43:51.730468 13939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:43:51.731848 13939 hybrid_clock.cc:648] HybridClock initialized: now 1781981031731756 us; error 87 us; skew 500 ppm
I20260620 18:43:51.732172 13939 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:43:51 dist-test-slave-d645 krb5kdc[13926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981031, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:43:51.735450 13939 init.cc:377] Logged in from keytab as kudu/127.23.115.62@KRBTEST.COM (short username kudu)
I20260620 18:43:51.736948 13939 webserver.cc:460] Webserver started at http://127.23.115.62:32969/ using document root <none> and password file <none>
I20260620 18:43:51.737629 13939 fs_manager.cc:359] Metadata directory not provided
I20260620 18:43:51.737754 13939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:43:51.738063 13939 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:43:51.741000 13939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980676253068-24012-0/minicluster-data/master-0/data/instance:
uuid: "bbe79c36f9de479bb753a665dcdbfd9e"
format_stamp: "Formatted at 2026-06-20 18:43:51 on dist-test-slave-d645"
server_key: "072103b61624f8283f83a396d31184ac"
server_key_iv: "6bdb8bfe50c60ebf2392a03e5eab6e29"
server_key_version: "encryptionkey@0"
I20260620 18:43:51.741777 13939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980676253068-24012-0/minicluster-data/master-0/wal/instance:
uuid: "bbe79c36f9de479bb753a665dcdbfd9e"
format_stamp: "Formatted at 2026-06-20 18:43:51 on dist-test-slave-d645"
server_key: "072103b61624f8283f83a396d31184ac"
server_key_iv: "6bdb8bfe50c60ebf2392a03e5eab6e29"
server_key_version: "encryptionkey@0"
I20260620 18:43:51.747511 13939 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 18:43:51.751031 13954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:51.752538 13939 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:43:51.752729 13939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980676253068-24012-0/minicluster-data/master-0/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980676253068-24012-0/minicluster-data/master-0/wal
uuid: "bbe79c36f9de479bb753a665dcdbfd9e"
format_stamp: "Formatted at 2026-06-20 18:43:51 on dist-test-slave-d645"
server_key: "072103b61624f8283f83a396d31184ac"
server_key_iv: "6bdb8bfe50c60ebf2392a03e5eab6e29"
server_key_version: "encryptionkey@0"
I20260620 18:43:51.753036 13939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980676253068-24012-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980676253068-24012-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980676253068-24012-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:43:51.773859 13939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:43:51.777930 13939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:43:51.778151 13939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:43:51.796422 13939 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.62:38193
I20260620 18:43:51.796471 14005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.62:38193 every 8 connection(s)
I20260620 18:43:51.797695 13939 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
I20260620 18:43:51.799238 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 13939
I20260620 18:43:51.799460 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980676253068-24012-0/minicluster-data/master-0/wal/instance
I20260620 18:43:51.800256 24012 external_mini_cluster.cc:1428] Setting key 2d0b299c3c0ed20215a989bcf93bae86
I20260620 18:43:51.802179 14006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:43:51.810847 14006 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.115.62" port: 38193 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:43:51 dist-test-slave-d645 krb5kdc[13926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981031, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.62@KRBTEST.COM for kudu/127.23.115.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.115.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.115.61@KRBTEST.COM" created.
I20260620 18:43:51.834218 14010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:51.814036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59764 (local address 127.23.115.62:38193)
0620 18:43:51.815754 (+ 1718us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:51.815760 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:51.815806 (+ 46us) server_negotiation.cc:407] Connection header received
0620 18:43:51.817612 (+ 1806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:51.817618 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:51.817853 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:51.818009 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:51.819490 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:51.820706 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:51.822615 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:51.823208 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:51.823369 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:51.827191 (+ 3822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:51.827199 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:51.827211 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:51.827245 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:51.830064 (+ 2819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:51.830735 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:51.830742 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:51.830744 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:51.830885 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:51.831287 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:51.831291 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:51.831293 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:51.831780 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:51.831961 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:51.832483 (+ 522us) server_negotiation.cc:299] Negotiation successful
0620 18:43:51.832911 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":154,"server-negotiator.queue_time_us":1534,"spinlock_wait_cycles":28928,"thread_start_us":1312,"threads_started":1}
I20260620 18:43:51.834250 14009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:51.814036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:38193 (local address 127.0.0.1:59764)
0620 18:43:51.814899 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:51.814980 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:51.817222 (+ 2242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:51.818061 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:51.818067 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:51.818791 (+ 724us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:43:51.819263 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:51.819287 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:51.820860 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:51.820866 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:51.822468 (+ 1602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:51.822483 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:51.823474 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:51.823486 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:51.823677 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:51.824330 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:43:51.824366 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:43:51.826983 (+ 2617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:43:51.830182 (+ 3199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:51.830193 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:51.830205 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:51.830610 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:51.831001 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:51.831006 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:51.831010 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:51.831182 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:51.831907 (+ 725us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:43:51.831914 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:43:51.832122 (+ 208us) client_negotiation.cc:769] Sending connection context
0620 18:43:51.832329 (+ 207us) client_negotiation.cc:240] Negotiation successful
0620 18:43:51.832795 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"mutex_wait_us":1,"thread_start_us":351,"threads_started":1}
I20260620 18:43:51.837302 14006 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.115.61" port: 38393 } has no permanent_uuid. Determining permanent_uuid...
I20260620 18:43:51.838339 14009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:51.837915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.61:38393 (local address 127.0.0.1:36382)
0620 18:43:51.838107 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:51.838220 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.61:38393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 18:43:51.838600 13955 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.115.61:38393: connect: Connection refused (error 111)
I20260620 18:43:51.840252 14009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:51.839787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.61:38393 (local address 127.0.0.1:36384)
0620 18:43:51.839959 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:51.840029 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.61:38393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260620 18:43:51.840477 14006 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.115.61:38393: Network error: Client connection negotiation failed: client connection to 127.23.115.61:38393: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.115.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.61.keytab.
Entry for principal kudu/127.23.115.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.61.keytab.
I20260620 18:43:51.862063 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980676253068-24012-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980676253068-24012-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980676253068-24012-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980676253068-24012-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.115.61:38393
--webserver_interface=127.23.115.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.61.keytab
--principal=kudu/127.23.115.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:37753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.115.62:38193,127.23.115.61:38393,127.23.115.60:44645
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:43:51.889851 14006 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.115.61" port: 38393 } attempt: 1
I20260620 18:43:51.891098 14009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:51.890592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.61:38393 (local address 127.0.0.1:36400)
0620 18:43:51.890810 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:51.890922 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.61:38393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260620 18:43:51.892133 14009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:51.891745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.61:38393 (local address 127.0.0.1:36408)
0620 18:43:51.891899 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:51.891985 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.61:38393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260620 18:43:51.892329 14006 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.115.61:38393: Network error: Client connection negotiation failed: client connection to 127.23.115.61:38393: connect: Connection refused (error 111)
I20260620 18:43:51.960564 14006 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.115.61" port: 38393 } attempt: 2
I20260620 18:43:51.961798 14009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:51.961256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.61:38393 (local address 127.0.0.1:36420)
0620 18:43:51.961546 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:51.961633 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.61:38393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
I20260620 18:43:51.962829 14009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:51.962475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.61:38393 (local address 127.0.0.1:36436)
0620 18:43:51.962608 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:51.962686 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.61:38393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 18:43:51.963038 14006 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.115.61:38393: Network error: Client connection negotiation failed: client connection to 127.23.115.61:38393: connect: Connection refused (error 111)
I20260620 18:43:52.054298 14006 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.115.61" port: 38393 } attempt: 3
I20260620 18:43:52.055552 14009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:52.055092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.61:38393 (local address 127.0.0.1:36452)
0620 18:43:52.055332 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:52.055421 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.61:38393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260620 18:43:52.056655 14009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:52.056235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.61:38393 (local address 127.0.0.1:36454)
0620 18:43:52.056413 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:52.056498 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.61:38393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260620 18:43:52.056874 14006 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.115.61:38393: Network error: Client connection negotiation failed: client connection to 127.23.115.61:38393: connect: Connection refused (error 111)
W20260620 18:43:52.185958 14013 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 18:43:52.186293 14013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:43:52.186403 14013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:43:52.186515 14013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:43:52.193780 14013 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:43:52.193885 14013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:43:52.193979 14013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:43:52.194046 14013 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:43:52.194128 14013 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:43:52.200196 14006 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.115.61" port: 38393 } attempt: 4
I20260620 18:43:52.201412 14009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:52.200873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.61:38393 (local address 127.0.0.1:36470)
0620 18:43:52.201194 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:52.201277 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.61:38393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184}
I20260620 18:43:52.202368 14009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:52.202036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.61:38393 (local address 127.0.0.1:36474)
0620 18:43:52.202175 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:52.202253 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.61:38393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260620 18:43:52.202545 14006 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.115.61:38393: Network error: Client connection negotiation failed: client connection to 127.23.115.61:38393: connect: Connection refused (error 111)
I20260620 18:43:52.204666 14013 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:37753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980676253068-24012-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980676253068-24012-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.23.115.62:38193,127.23.115.61:38393,127.23.115.60:44645
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.61.keytab
--principal=kudu/127.23.115.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.115.61:38393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980676253068-24012-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.23.115.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980676253068-24012-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:43:52.205911 14013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:43:52.207167 14013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:43:52.218302 14018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:52.218253 14021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:52.218489 14019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:52.219668 14013 server_base.cc:1034] running on GCE node
I20260620 18:43:52.220394 14013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:43:52.221733 14013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:43:52.222990 14013 hybrid_clock.cc:648] HybridClock initialized: now 1781981032222909 us; error 29 us; skew 500 ppm
I20260620 18:43:52.223383 14013 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:43:52 dist-test-slave-d645 krb5kdc[13926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981032, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:43:52.226471 14013 init.cc:377] Logged in from keytab as kudu/127.23.115.61@KRBTEST.COM (short username kudu)
I20260620 18:43:52.228116 14013 webserver.cc:460] Webserver started at http://127.23.115.61:35227/ using document root <none> and password file <none>
I20260620 18:43:52.228768 14013 fs_manager.cc:359] Metadata directory not provided
I20260620 18:43:52.228863 14013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:43:52.229120 14013 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:43:52.232023 14013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980676253068-24012-0/minicluster-data/master-1/data/instance:
uuid: "c230608d2fc847198dcce513367e58b7"
format_stamp: "Formatted at 2026-06-20 18:43:52 on dist-test-slave-d645"
server_key: "9c0ebaa9abe37e7a8f05b2536d9249c0"
server_key_iv: "f7b10d94f1dbc10c04e496f5c0682a2a"
server_key_version: "encryptionkey@0"
I20260620 18:43:52.232744 14013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980676253068-24012-0/minicluster-data/master-1/wal/instance:
uuid: "c230608d2fc847198dcce513367e58b7"
format_stamp: "Formatted at 2026-06-20 18:43:52 on dist-test-slave-d645"
server_key: "9c0ebaa9abe37e7a8f05b2536d9249c0"
server_key_iv: "f7b10d94f1dbc10c04e496f5c0682a2a"
server_key_version: "encryptionkey@0"
I20260620 18:43:52.237761 14013 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 18:43:52.241196 14028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:52.242858 14013 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260620 18:43:52.243038 14013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980676253068-24012-0/minicluster-data/master-1/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980676253068-24012-0/minicluster-data/master-1/wal
uuid: "c230608d2fc847198dcce513367e58b7"
format_stamp: "Formatted at 2026-06-20 18:43:52 on dist-test-slave-d645"
server_key: "9c0ebaa9abe37e7a8f05b2536d9249c0"
server_key_iv: "f7b10d94f1dbc10c04e496f5c0682a2a"
server_key_version: "encryptionkey@0"
I20260620 18:43:52.243232 14013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980676253068-24012-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980676253068-24012-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980676253068-24012-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:43:52.270668 14013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:43:52.275939 14013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:43:52.276180 14013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:43:52.294080 14013 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.61:38393
I20260620 18:43:52.294086 14079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.61:38393 every 8 connection(s)
I20260620 18:43:52.295439 14013 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980676253068-24012-0/minicluster-data/master-1/data/info.pb
I20260620 18:43:52.296085 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 14013
I20260620 18:43:52.296263 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980676253068-24012-0/minicluster-data/master-1/wal/instance
I20260620 18:43:52.296792 24012 external_mini_cluster.cc:1428] Setting key b624908381c95450a52f987947b863ea
I20260620 18:43:52.299880 14080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:43:52.308033 14080 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.115.62" port: 38193 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:43:52 dist-test-slave-d645 krb5kdc[13926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981032, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.61@KRBTEST.COM for kudu/127.23.115.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.115.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.115.60@KRBTEST.COM" created.
I20260620 18:43:52.330675 14010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:52.310508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59772 (local address 127.23.115.62:38193)
0620 18:43:52.310719 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:52.310725 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:52.311901 (+ 1176us) server_negotiation.cc:407] Connection header received
0620 18:43:52.314265 (+ 2364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:52.314272 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:52.314343 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:52.314436 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:52.316343 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:52.317005 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:52.319368 (+ 2363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:52.319803 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:52.319914 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:52.324241 (+ 4327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:52.324249 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:52.324253 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:52.324307 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:52.327488 (+ 3181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:52.328393 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:52.328397 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:52.328400 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:52.328525 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:52.328986 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:52.328990 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:52.328993 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:52.329319 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:52.329454 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:52.330257 (+ 803us) server_negotiation.cc:299] Negotiation successful
0620 18:43:52.330431 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 18:43:52.331945 14083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:52.310962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:38193 (local address 127.0.0.1:59772)
0620 18:43:52.311729 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:52.311773 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:52.313951 (+ 2178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:52.314665 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:52.314685 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:52.315412 (+ 727us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:43:52.316128 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:52.316151 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:52.317156 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:52.317162 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:52.319186 (+ 2024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:52.319205 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:52.320051 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:52.320058 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:52.320347 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:52.321067 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:43:52.321097 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:43:52.323824 (+ 2727us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:43:52.327647 (+ 3823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:52.327656 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:52.327674 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:52.328263 (+ 589us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:52.328623 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:52.328628 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:52.328630 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:52.328843 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:52.329442 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:43:52.329450 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:43:52.329866 (+ 416us) client_negotiation.cc:769] Sending connection context
0620 18:43:52.330142 (+ 276us) client_negotiation.cc:240] Negotiation successful
0620 18:43:52.330636 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":288,"threads_started":1}
I20260620 18:43:52.333706 14080 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.115.61" port: 38393 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:43:52 dist-test-slave-d645 krb5kdc[13926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981032, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.61@KRBTEST.COM for kudu/127.23.115.61@KRBTEST.COM
I20260620 18:43:52.349361 14084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:52.334737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36476 (local address 127.23.115.61:38393)
0620 18:43:52.335332 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:52.335338 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:52.335363 (+ 25us) server_negotiation.cc:407] Connection header received
0620 18:43:52.335405 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:52.335411 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:52.335580 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:52.335691 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:52.336522 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:52.338029 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:52.339288 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:52.339802 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:52.339914 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:52.342985 (+ 3071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:52.342998 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:52.343012 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:52.343045 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:52.345946 (+ 2901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:52.347109 (+ 1163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:52.347116 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:52.347118 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:52.347288 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:52.347729 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:52.347733 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:52.347735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:52.348086 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:52.348259 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:52.348761 (+ 502us) server_negotiation.cc:299] Negotiation successful
0620 18:43:52.349048 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":56,"server-negotiator.queue_time_us":453,"thread_start_us":311,"threads_started":1}
I20260620 18:43:52.349391 14083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:52.334516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.61:38393 (local address 127.0.0.1:36476)
0620 18:43:52.334681 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:52.334700 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:52.334799 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:52.335735 (+ 936us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:52.335743 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:52.336182 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:43:52.336355 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:52.336366 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:52.338178 (+ 1812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:52.338182 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:52.339159 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:52.339172 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:52.339935 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:52.339940 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:52.340113 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:52.340682 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:43:52.340699 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:43:52.342773 (+ 2074us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:43:52.346101 (+ 3328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:52.346107 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:52.346111 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:52.346971 (+ 860us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:52.347406 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:52.347411 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:52.347416 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:52.347595 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:52.348244 (+ 649us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:43:52.348250 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:43:52.348417 (+ 167us) client_negotiation.cc:769] Sending connection context
0620 18:43:52.348593 (+ 176us) client_negotiation.cc:240] Negotiation successful
0620 18:43:52.348925 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 18:43:52.352536 14080 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.115.60" port: 44645 } has no permanent_uuid. Determining permanent_uuid...
I20260620 18:43:52.353989 14083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:52.353352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.60:44645 (local address 127.0.0.1:56122)
0620 18:43:52.353643 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:52.353809 (+ 166us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.60:44645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167}
W20260620 18:43:52.354271 14032 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.115.60:44645: connect: Connection refused (error 111)
I20260620 18:43:52.355890 14083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:52.355543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.60:44645 (local address 127.0.0.1:56128)
0620 18:43:52.355698 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:52.355785 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.60:44645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 18:43:52.356077 14080 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.115.60:44645: Network error: Client connection negotiation failed: client connection to 127.23.115.60:44645: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.115.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.60.keytab.
Entry for principal kudu/127.23.115.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.60.keytab.
I20260620 18:43:52.358543 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980676253068-24012-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980676253068-24012-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980676253068-24012-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980676253068-24012-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.115.60:44645
--webserver_interface=127.23.115.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.60.keytab
--principal=kudu/127.23.115.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:37753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.115.62:38193,127.23.115.61:38393,127.23.115.60:44645
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:43:52.405505 14080 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.115.60" port: 44645 } attempt: 1
I20260620 18:43:52.407003 14083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:52.406514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.60:44645 (local address 127.0.0.1:56130)
0620 18:43:52.406709 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:52.406802 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.60:44645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260620 18:43:52.408138 14083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:52.407742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.60:44645 (local address 127.0.0.1:56136)
0620 18:43:52.407928 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:52.408020 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.60:44645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260620 18:43:52.408360 14080 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.115.60:44645: Network error: Client connection negotiation failed: client connection to 127.23.115.60:44645: connect: Connection refused (error 111)
I20260620 18:43:52.476583 14080 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.115.60" port: 44645 } attempt: 2
I20260620 18:43:52.477732 14083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:52.477265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.60:44645 (local address 127.0.0.1:56142)
0620 18:43:52.477465 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:52.477553 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.60:44645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260620 18:43:52.478775 14083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:52.478423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.60:44645 (local address 127.0.0.1:56144)
0620 18:43:52.478582 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:52.478650 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.60:44645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260620 18:43:52.479007 14080 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.115.60:44645: Network error: Client connection negotiation failed: client connection to 127.23.115.60:44645: connect: Connection refused (error 111)
I20260620 18:43:52.501838 14006 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.115.61" port: 38393 } attempt: 5
Jun 20 18:43:52 dist-test-slave-d645 krb5kdc[13926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981031, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.62@KRBTEST.COM for kudu/127.23.115.61@KRBTEST.COM
I20260620 18:43:52.514560 14009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:52.502585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.61:38393 (local address 127.0.0.1:36480)
0620 18:43:52.502763 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:52.502782 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:52.502997 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:52.503315 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:52.503319 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:52.503708 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:43:52.503899 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:52.503909 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:52.504867 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:52.504873 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:52.505710 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:52.505721 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:52.506429 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:52.506435 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:52.506627 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:52.507295 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:43:52.507318 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:43:52.509514 (+ 2196us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:43:52.512325 (+ 2811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:52.512331 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:52.512335 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:52.512681 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:52.513003 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:52.513008 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:52.513013 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:52.513139 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:52.513625 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:43:52.513630 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:43:52.513798 (+ 168us) client_negotiation.cc:769] Sending connection context
0620 18:43:52.513920 (+ 122us) client_negotiation.cc:240] Negotiation successful
0620 18:43:52.514196 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260620 18:43:52.514556 14084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:52.502769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36480 (local address 127.23.115.61:38393)
0620 18:43:52.503005 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:52.503011 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:52.503032 (+ 21us) server_negotiation.cc:407] Connection header received
0620 18:43:52.503097 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:52.503103 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:52.503183 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:52.503263 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:52.504044 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:52.504738 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:52.505853 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:52.506275 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:52.506406 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:52.509696 (+ 3290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:52.509701 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:52.509704 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:52.509781 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:52.512203 (+ 2422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:52.512787 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:52.512793 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:52.512795 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:52.512905 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:52.513235 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:52.513240 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:52.513244 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:52.513516 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:52.513644 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:52.514025 (+ 381us) server_negotiation.cc:299] Negotiation successful
0620 18:43:52.514233 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 18:43:52.515679 14006 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.115.60" port: 44645 } has no permanent_uuid. Determining permanent_uuid...
I20260620 18:43:52.516872 14009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:52.516418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.60:44645 (local address 127.0.0.1:56146)
0620 18:43:52.516605 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:52.516700 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.60:44645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260620 18:43:52.517896 14009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:52.517528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.60:44645 (local address 127.0.0.1:56148)
0620 18:43:52.517671 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:52.517767 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.60:44645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 18:43:52.518111 14006 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.115.60:44645: Network error: Client connection negotiation failed: client connection to 127.23.115.60:44645: connect: Connection refused (error 111)
I20260620 18:43:52.569367 14006 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.115.60" port: 44645 } attempt: 1
I20260620 18:43:52.570236 14080 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.115.60" port: 44645 } attempt: 3
I20260620 18:43:52.570595 14009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:52.570067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.60:44645 (local address 127.0.0.1:56158)
0620 18:43:52.570282 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:52.570398 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.60:44645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260620 18:43:52.571255 14083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:52.570769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.60:44645 (local address 127.0.0.1:56162)
0620 18:43:52.570997 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:52.571092 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.60:44645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260620 18:43:52.571755 14009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:52.571330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.60:44645 (local address 127.0.0.1:56174)
0620 18:43:52.571534 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:52.571622 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.60:44645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
W20260620 18:43:52.571973 14006 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.115.60:44645: Network error: Client connection negotiation failed: client connection to 127.23.115.60:44645: connect: Connection refused (error 111)
I20260620 18:43:52.572364 14083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:52.571845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.60:44645 (local address 127.0.0.1:56180)
0620 18:43:52.572135 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:52.572227 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.60:44645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186}
W20260620 18:43:52.572582 14080 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.115.60:44645: Network error: Client connection negotiation failed: client connection to 127.23.115.60:44645: connect: Connection refused (error 111)
I20260620 18:43:52.640218 14006 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.115.60" port: 44645 } attempt: 2
I20260620 18:43:52.641398 14009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:52.640946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.60:44645 (local address 127.0.0.1:56194)
0620 18:43:52.641125 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:52.641229 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.60:44645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260620 18:43:52.642351 14009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:52.642034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.60:44645 (local address 127.0.0.1:56202)
0620 18:43:52.642185 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:52.642256 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.60:44645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 18:43:52.642544 14006 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.115.60:44645: Network error: Client connection negotiation failed: client connection to 127.23.115.60:44645: connect: Connection refused (error 111)
W20260620 18:43:52.676295 14087 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 18:43:52.676651 14087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:43:52.676746 14087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:43:52.676864 14087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:43:52.684302 14087 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:43:52.684415 14087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:43:52.684499 14087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:43:52.684566 14087 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:43:52.684652 14087 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:43:52.694422 14087 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:37753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980676253068-24012-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980676253068-24012-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.23.115.62:38193,127.23.115.61:38393,127.23.115.60:44645
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.60.keytab
--principal=kudu/127.23.115.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.115.60:44645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980676253068-24012-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.23.115.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980676253068-24012-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:43:52.695686 14087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:43:52.696897 14087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:43:52.707967 14095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:52.708069 14093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:52.708680 14092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:52.709441 14087 server_base.cc:1034] running on GCE node
I20260620 18:43:52.710263 14087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:43:52.711879 14087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:43:52.713096 14087 hybrid_clock.cc:648] HybridClock initialized: now 1781981032713015 us; error 47 us; skew 500 ppm
I20260620 18:43:52.713405 14087 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:43:52 dist-test-slave-d645 krb5kdc[13926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981032, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:43:52.715813 14080 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.115.60" port: 44645 } attempt: 4
I20260620 18:43:52.716468 14087 init.cc:377] Logged in from keytab as kudu/127.23.115.60@KRBTEST.COM (short username kudu)
I20260620 18:43:52.716876 14083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:52.716320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.60:44645 (local address 127.0.0.1:56216)
0620 18:43:52.716567 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:52.716671 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.60:44645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260620 18:43:52.717832 14083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:52.717448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.60:44645 (local address 127.0.0.1:56230)
0620 18:43:52.717603 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:52.717688 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.60:44645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260620 18:43:52.718075 14087 webserver.cc:460] Webserver started at http://127.23.115.60:36295/ using document root <none> and password file <none>
W20260620 18:43:52.718097 14080 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.115.60:44645: Network error: Client connection negotiation failed: client connection to 127.23.115.60:44645: connect: Connection refused (error 111)
I20260620 18:43:52.718717 14087 fs_manager.cc:359] Metadata directory not provided
I20260620 18:43:52.718809 14087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:43:52.719146 14087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:43:52.721973 14087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980676253068-24012-0/minicluster-data/master-2/data/instance:
uuid: "a6400bf25e5745589c4cd0e78dc93f5a"
format_stamp: "Formatted at 2026-06-20 18:43:52 on dist-test-slave-d645"
server_key: "b6b0c94f95ee73d78b20814f833b7a93"
server_key_iv: "f6508dcafb875876d8ab0b8e74d4ee90"
server_key_version: "encryptionkey@0"
I20260620 18:43:52.722710 14087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980676253068-24012-0/minicluster-data/master-2/wal/instance:
uuid: "a6400bf25e5745589c4cd0e78dc93f5a"
format_stamp: "Formatted at 2026-06-20 18:43:52 on dist-test-slave-d645"
server_key: "b6b0c94f95ee73d78b20814f833b7a93"
server_key_iv: "f6508dcafb875876d8ab0b8e74d4ee90"
server_key_version: "encryptionkey@0"
I20260620 18:43:52.727584 14087 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260620 18:43:52.730783 14102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:52.732051 14087 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 18:43:52.732203 14087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980676253068-24012-0/minicluster-data/master-2/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980676253068-24012-0/minicluster-data/master-2/wal
uuid: "a6400bf25e5745589c4cd0e78dc93f5a"
format_stamp: "Formatted at 2026-06-20 18:43:52 on dist-test-slave-d645"
server_key: "b6b0c94f95ee73d78b20814f833b7a93"
server_key_iv: "f6508dcafb875876d8ab0b8e74d4ee90"
server_key_version: "encryptionkey@0"
I20260620 18:43:52.732409 14087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980676253068-24012-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980676253068-24012-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980676253068-24012-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:43:52.748728 14006 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.115.60" port: 44645 } attempt: 3
I20260620 18:43:52.749848 14009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:52.749364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.60:44645 (local address 127.0.0.1:56238)
0620 18:43:52.749578 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:52.749674 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.60:44645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260620 18:43:52.750780 14009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:52.750408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.60:44645 (local address 127.0.0.1:56248)
0620 18:43:52.750589 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:52.750667 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.60:44645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260620 18:43:52.751021 14006 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.115.60:44645: Network error: Client connection negotiation failed: client connection to 127.23.115.60:44645: connect: Connection refused (error 111)
I20260620 18:43:52.774112 14087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:43:52.778425 14087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:43:52.778733 14087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:43:52.796458 14087 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.60:44645
I20260620 18:43:52.796507 14153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.60:44645 every 8 connection(s)
I20260620 18:43:52.797689 14087 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980676253068-24012-0/minicluster-data/master-2/data/info.pb
I20260620 18:43:52.801898 14154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:43:52.802392 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 14087
I20260620 18:43:52.802556 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980676253068-24012-0/minicluster-data/master-2/wal/instance
I20260620 18:43:52.803030 24012 external_mini_cluster.cc:1428] Setting key 9c9ae365bfc459fda10aab65a91150b9
I20260620 18:43:52.812243 14154 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.115.62" port: 38193 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:43:52 dist-test-slave-d645 krb5kdc[13926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.115.60@KRBTEST.COM
Jun 20 18:43:52 dist-test-slave-d645 krb5kdc[13926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981032, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.60@KRBTEST.COM for kudu/127.23.115.62@KRBTEST.COM
I20260620 18:43:52.826210 14157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:52.804713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56262 (local address 127.23.115.60:44645)
0620 18:43:52.805528 (+ 815us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:52.805539 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:52.805579 (+ 40us) server_negotiation.cc:407] Connection header received
0620 18:43:52.808066 (+ 2487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:52.808087 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:52.808453 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:52.808805 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:52.810204 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:52.811500 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:52.812711 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:52.813407 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:52.813563 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:52.818224 (+ 4661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:52.818235 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:52.818248 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:52.818294 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:52.822119 (+ 3825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:52.822758 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:52.822764 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:52.822769 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:52.822966 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:52.823388 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:52.823391 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:52.823393 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:52.823901 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:52.824169 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:52.824567 (+ 398us) server_negotiation.cc:299] Negotiation successful
0620 18:43:52.824945 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":491,"thread_start_us":294,"threads_started":1}
I20260620 18:43:52.829672 14010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:52.814709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59788 (local address 127.23.115.62:38193)
0620 18:43:52.814968 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:52.814975 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:52.815485 (+ 510us) server_negotiation.cc:407] Connection header received
0620 18:43:52.815658 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:52.815664 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:52.815756 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:52.815867 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:52.816867 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:52.817809 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:52.819737 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:52.820277 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:52.820374 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:52.823827 (+ 3453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:52.823831 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:52.823834 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:52.823865 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:52.826797 (+ 2932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:52.827632 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:52.827637 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:52.827641 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:52.827793 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:52.828145 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:52.828150 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:52.828154 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:52.828397 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:52.828533 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:52.829118 (+ 585us) server_negotiation.cc:299] Negotiation successful
0620 18:43:52.829324 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 18:43:52.829684 14158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:52.814620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:38193 (local address 127.0.0.1:59788)
0620 18:43:52.815347 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:52.815385 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:52.815530 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:52.815932 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:52.815937 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:52.816481 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:43:52.816713 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:52.816721 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:52.817974 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:52.817978 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:52.819555 (+ 1577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:52.819572 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:52.820462 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:52.820476 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:52.820649 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:52.821196 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:43:52.821233 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:43:52.823646 (+ 2413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:43:52.826963 (+ 3317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:52.826971 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:52.826977 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:52.827500 (+ 523us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:52.827905 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:52.827907 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:52.827910 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:52.828033 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:52.828525 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:43:52.828532 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:43:52.828711 (+ 179us) client_negotiation.cc:769] Sending connection context
0620 18:43:52.828969 (+ 258us) client_negotiation.cc:240] Negotiation successful
0620 18:43:52.829270 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":374,"threads_started":1}
I20260620 18:43:52.831194 14154 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.115.61" port: 38393 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:43:52 dist-test-slave-d645 krb5kdc[13926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981032, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.60@KRBTEST.COM for kudu/127.23.115.61@KRBTEST.COM
Jun 20 18:43:52 dist-test-slave-d645 krb5kdc[13926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.115.62@KRBTEST.COM
I20260620 18:43:52.845504 14158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:52.831768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.61:38393 (local address 127.0.0.1:36494)
0620 18:43:52.831979 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:52.831995 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:52.832081 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:52.832750 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:52.832755 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:52.833183 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:43:52.833336 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:52.833346 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:52.834678 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:52.834683 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:52.835802 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:52.835816 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:52.836690 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:52.836696 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:52.836883 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:52.837591 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:43:52.837616 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:43:52.840076 (+ 2460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:43:52.843095 (+ 3019us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:52.843101 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:52.843105 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:52.843465 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:52.843827 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:52.843832 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:52.843836 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:52.843968 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:52.844464 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:43:52.844469 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:43:52.844631 (+ 162us) client_negotiation.cc:769] Sending connection context
0620 18:43:52.844753 (+ 122us) client_negotiation.cc:240] Negotiation successful
0620 18:43:52.845058 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123}
I20260620 18:43:52.845530 14084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:52.831961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36494 (local address 127.23.115.61:38393)
0620 18:43:52.832147 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:52.832153 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:52.832171 (+ 18us) server_negotiation.cc:407] Connection header received
0620 18:43:52.832229 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:52.832235 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:52.832305 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:52.832393 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:52.833499 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:52.834503 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:52.835966 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:52.836547 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:52.836642 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:52.840260 (+ 3618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:52.840265 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:52.840268 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:52.840302 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:52.842979 (+ 2677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:52.843574 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:52.843580 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:52.843584 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:52.843711 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:52.844068 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:52.844073 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:52.844076 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:52.844342 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:52.844481 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:52.844865 (+ 384us) server_negotiation.cc:299] Negotiation successful
0620 18:43:52.845073 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 18:43:52.846187 14010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:52.830416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59804 (local address 127.23.115.62:38193)
0620 18:43:52.830563 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:52.830568 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:52.830584 (+ 16us) server_negotiation.cc:407] Connection header received
0620 18:43:52.830744 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:52.830749 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:52.830820 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:52.830913 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:52.832422 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:52.833474 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:52.835142 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:52.835751 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:52.835878 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:52.840831 (+ 4953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:52.840836 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:52.840839 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:52.840869 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:52.843325 (+ 2456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:52.844075 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:52.844079 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:52.844081 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:52.844206 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:52.844678 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:52.844683 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:52.844686 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:52.844962 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:52.845231 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:52.845517 (+ 286us) server_negotiation.cc:299] Negotiation successful
0620 18:43:52.845892 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 18:43:52.846628 14154 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.115.60" port: 44645 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:43:52 dist-test-slave-d645 krb5kdc[13926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981032, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.60@KRBTEST.COM for kudu/127.23.115.60@KRBTEST.COM
Jun 20 18:43:52 dist-test-slave-d645 krb5kdc[13926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.115.61@KRBTEST.COM
I20260620 18:43:52.858450 14158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:52.847522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.60:44645 (local address 127.0.0.1:56270)
0620 18:43:52.847673 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:52.847691 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:52.847903 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:52.848162 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:52.848166 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:52.848509 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:43:52.848630 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:52.848637 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:52.849512 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:52.849517 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:52.850329 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:52.850340 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:52.850990 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:52.850996 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:52.851162 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:52.851631 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:43:52.851646 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:43:52.853647 (+ 2001us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:43:52.856206 (+ 2559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:52.856212 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:52.856216 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:52.856632 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:52.856927 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:52.856932 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:52.856936 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:52.857049 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:52.857542 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:43:52.857546 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:43:52.857677 (+ 131us) client_negotiation.cc:769] Sending connection context
0620 18:43:52.857795 (+ 118us) client_negotiation.cc:240] Negotiation successful
0620 18:43:52.858060 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260620 18:43:52.858513 14157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:52.847750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56270 (local address 127.23.115.60:44645)
0620 18:43:52.847867 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:52.847872 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:52.847888 (+ 16us) server_negotiation.cc:407] Connection header received
0620 18:43:52.848002 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:52.848007 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:52.848067 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:52.848137 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:52.848748 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:52.849404 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:52.850444 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:52.850833 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:52.850962 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:52.853809 (+ 2847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:52.853813 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:52.853817 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:52.853842 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:52.856092 (+ 2250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:52.856724 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:52.856730 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:52.856733 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:52.856839 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:52.857136 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:52.857141 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:52.857144 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:52.857447 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:52.857565 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:52.857889 (+ 324us) server_negotiation.cc:299] Negotiation successful
0620 18:43:52.858218 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":115,"server-negotiator.queue_time_us":31}
I20260620 18:43:52.860553 14084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:52.847778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36498 (local address 127.23.115.61:38393)
0620 18:43:52.847921 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:52.847926 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:52.848008 (+ 82us) server_negotiation.cc:407] Connection header received
0620 18:43:52.848163 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:52.848168 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:52.848224 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:52.848298 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:52.849432 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:52.850114 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:52.851305 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:52.851803 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:52.851916 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:52.855766 (+ 3850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:52.855770 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:52.855773 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:52.855802 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:52.858163 (+ 2361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:52.858831 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:52.858836 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:52.858838 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:52.859022 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:52.859403 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:52.859406 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:52.859408 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:52.859693 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:52.859960 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:52.860178 (+ 218us) server_negotiation.cc:299] Negotiation successful
0620 18:43:52.860337 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 18:43:52.864322 14154 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6400bf25e5745589c4cd0e78dc93f5a: Bootstrap starting.
I20260620 18:43:52.865305 24012 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:43:52.868322 14154 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a6400bf25e5745589c4cd0e78dc93f5a: Neither blocks nor log segments found. Creating new log.
I20260620 18:43:52.869334 14154 log.cc:824] T 00000000000000000000000000000000 P a6400bf25e5745589c4cd0e78dc93f5a: Log is configured to *not* fsync() on all Append() calls
I20260620 18:43:52.868672 13981 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "bbe79c36f9de479bb753a665dcdbfd9e"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59804
I20260620 18:43:52.872143 14154 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6400bf25e5745589c4cd0e78dc93f5a: No bootstrap required, opened a new log
I20260620 18:43:52.879357 14154 raft_consensus.cc:348] T 00000000000000000000000000000000 P a6400bf25e5745589c4cd0e78dc93f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe79c36f9de479bb753a665dcdbfd9e" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 38193 } } peers { permanent_uuid: "c230608d2fc847198dcce513367e58b7" member_type: VOTER last_known_addr { host: "127.23.115.61" port: 38393 } } peers { permanent_uuid: "a6400bf25e5745589c4cd0e78dc93f5a" member_type: VOTER last_known_addr { host: "127.23.115.60" port: 44645 } }
I20260620 18:43:52.879688 14154 raft_consensus.cc:729] T 00000000000000000000000000000000 P a6400bf25e5745589c4cd0e78dc93f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6400bf25e5745589c4cd0e78dc93f5a, State: Initialized, Role: FOLLOWER
I20260620 18:43:52.880121 14154 consensus_queue.cc:260] T 00000000000000000000000000000000 P a6400bf25e5745589c4cd0e78dc93f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe79c36f9de479bb753a665dcdbfd9e" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 38193 } } peers { permanent_uuid: "c230608d2fc847198dcce513367e58b7" member_type: VOTER last_known_addr { host: "127.23.115.61" port: 38393 } } peers { permanent_uuid: "a6400bf25e5745589c4cd0e78dc93f5a" member_type: VOTER last_known_addr { host: "127.23.115.60" port: 44645 } }
I20260620 18:43:52.880978 14160 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6400bf25e5745589c4cd0e78dc93f5a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe79c36f9de479bb753a665dcdbfd9e" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 38193 } } peers { permanent_uuid: "c230608d2fc847198dcce513367e58b7" member_type: VOTER last_known_addr { host: "127.23.115.61" port: 38393 } } peers { permanent_uuid: "a6400bf25e5745589c4cd0e78dc93f5a" member_type: VOTER last_known_addr { host: "127.23.115.60" port: 44645 } } }
I20260620 18:43:52.881345 14160 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6400bf25e5745589c4cd0e78dc93f5a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:43:52.882032 14154 sys_catalog.cc:564] T 00000000000000000000000000000000 P a6400bf25e5745589c4cd0e78dc93f5a [sys.catalog]: configured and running, proceeding with master startup.
W20260620 18:43:52.890384 14171 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a6400bf25e5745589c4cd0e78dc93f5a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 18:43:52.890493 14171 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 18:43:52.888944 14087 master_runner.cc:186] Error getting master registration for 127.23.115.62:38193: OK, instance_id { permanent_uuid: "bbe79c36f9de479bb753a665dcdbfd9e" instance_seqno: 1781981031791154 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 18:43:52.899682 14087 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 18:43:52.928170 14006 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.115.60" port: 44645 } attempt: 4
Jun 20 18:43:52 dist-test-slave-d645 krb5kdc[13926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981031, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.62@KRBTEST.COM for kudu/127.23.115.60@KRBTEST.COM
I20260620 18:43:52.942734 14009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:52.928947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.60:44645 (local address 127.0.0.1:56278)
0620 18:43:52.929134 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:52.929153 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:52.929297 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:52.929649 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:52.929655 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:52.930062 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:43:52.930240 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:52.930258 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:52.931408 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:52.931415 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:52.932360 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:52.932372 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:52.933122 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:52.933128 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:52.933325 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:52.933898 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:43:52.933915 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:43:52.936281 (+ 2366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:43:52.940114 (+ 3833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:52.940119 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:52.940124 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:52.940610 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:52.940998 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:52.941001 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:52.941004 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:52.941126 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:52.941739 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:43:52.941744 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:43:52.941927 (+ 183us) client_negotiation.cc:769] Sending connection context
0620 18:43:52.942069 (+ 142us) client_negotiation.cc:240] Negotiation successful
0620 18:43:52.942348 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260620 18:43:52.942771 14157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:52.929161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56278 (local address 127.23.115.60:44645)
0620 18:43:52.929368 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:52.929373 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:52.929391 (+ 18us) server_negotiation.cc:407] Connection header received
0620 18:43:52.929455 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:52.929459 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:52.929518 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:52.929599 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:52.930395 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:52.931267 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:52.932499 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:52.932985 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:52.933091 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:52.936460 (+ 3369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:52.936467 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:52.936472 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:52.936511 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:52.939979 (+ 3468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:52.940739 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:52.940744 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:52.940748 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:52.940890 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:52.941232 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:52.941237 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:52.941240 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:52.941609 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:52.941776 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:52.942200 (+ 424us) server_negotiation.cc:299] Negotiation successful
0620 18:43:52.942420 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 18:43:52.948882 14006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbe79c36f9de479bb753a665dcdbfd9e: Bootstrap starting.
I20260620 18:43:52.952797 14006 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bbe79c36f9de479bb753a665dcdbfd9e: Neither blocks nor log segments found. Creating new log.
I20260620 18:43:52.954036 14006 log.cc:824] T 00000000000000000000000000000000 P bbe79c36f9de479bb753a665dcdbfd9e: Log is configured to *not* fsync() on all Append() calls
I20260620 18:43:52.957177 14006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbe79c36f9de479bb753a665dcdbfd9e: No bootstrap required, opened a new log
I20260620 18:43:52.964706 14006 raft_consensus.cc:348] T 00000000000000000000000000000000 P bbe79c36f9de479bb753a665dcdbfd9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe79c36f9de479bb753a665dcdbfd9e" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 38193 } } peers { permanent_uuid: "c230608d2fc847198dcce513367e58b7" member_type: VOTER last_known_addr { host: "127.23.115.61" port: 38393 } } peers { permanent_uuid: "a6400bf25e5745589c4cd0e78dc93f5a" member_type: VOTER last_known_addr { host: "127.23.115.60" port: 44645 } }
I20260620 18:43:52.965106 14006 raft_consensus.cc:729] T 00000000000000000000000000000000 P bbe79c36f9de479bb753a665dcdbfd9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbe79c36f9de479bb753a665dcdbfd9e, State: Initialized, Role: FOLLOWER
I20260620 18:43:52.965608 14006 consensus_queue.cc:260] T 00000000000000000000000000000000 P bbe79c36f9de479bb753a665dcdbfd9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe79c36f9de479bb753a665dcdbfd9e" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 38193 } } peers { permanent_uuid: "c230608d2fc847198dcce513367e58b7" member_type: VOTER last_known_addr { host: "127.23.115.61" port: 38393 } } peers { permanent_uuid: "a6400bf25e5745589c4cd0e78dc93f5a" member_type: VOTER last_known_addr { host: "127.23.115.60" port: 44645 } }
I20260620 18:43:52.966598 14174 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbe79c36f9de479bb753a665dcdbfd9e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe79c36f9de479bb753a665dcdbfd9e" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 38193 } } peers { permanent_uuid: "c230608d2fc847198dcce513367e58b7" member_type: VOTER last_known_addr { host: "127.23.115.61" port: 38393 } } peers { permanent_uuid: "a6400bf25e5745589c4cd0e78dc93f5a" member_type: VOTER last_known_addr { host: "127.23.115.60" port: 44645 } } }
I20260620 18:43:52.967016 14174 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbe79c36f9de479bb753a665dcdbfd9e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:43:52.967849 14006 sys_catalog.cc:564] T 00000000000000000000000000000000 P bbe79c36f9de479bb753a665dcdbfd9e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:43:52.967939 13981 raft_consensus.cc:482] T 00000000000000000000000000000000 P bbe79c36f9de479bb753a665dcdbfd9e [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 18:43:52.968132 13981 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bbe79c36f9de479bb753a665dcdbfd9e [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:43:52.970783 13981 raft_consensus.cc:504] T 00000000000000000000000000000000 P bbe79c36f9de479bb753a665dcdbfd9e [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe79c36f9de479bb753a665dcdbfd9e" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 38193 } } peers { permanent_uuid: "c230608d2fc847198dcce513367e58b7" member_type: VOTER last_known_addr { host: "127.23.115.61" port: 38393 } } peers { permanent_uuid: "a6400bf25e5745589c4cd0e78dc93f5a" member_type: VOTER last_known_addr { host: "127.23.115.60" port: 44645 } }
I20260620 18:43:52.973165 14129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bbe79c36f9de479bb753a665dcdbfd9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a6400bf25e5745589c4cd0e78dc93f5a"
I20260620 18:43:52.973577 14129 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a6400bf25e5745589c4cd0e78dc93f5a [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:43:52.975991 14129 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a6400bf25e5745589c4cd0e78dc93f5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbe79c36f9de479bb753a665dcdbfd9e in term 1.
I20260620 18:43:52.976720 13958 leader_election.cc:302] T 00000000000000000000000000000000 P bbe79c36f9de479bb753a665dcdbfd9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6400bf25e5745589c4cd0e78dc93f5a, bbe79c36f9de479bb753a665dcdbfd9e; no voters:
I20260620 18:43:52.976720 13981 leader_election.cc:288] T 00000000000000000000000000000000 P bbe79c36f9de479bb753a665dcdbfd9e [CANDIDATE]: Term 1 election: Requested vote from peers c230608d2fc847198dcce513367e58b7 (127.23.115.61:38393), a6400bf25e5745589c4cd0e78dc93f5a (127.23.115.60:44645)
I20260620 18:43:52.977295 14174 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bbe79c36f9de479bb753a665dcdbfd9e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:43:52.977936 14174 raft_consensus.cc:686] T 00000000000000000000000000000000 P bbe79c36f9de479bb753a665dcdbfd9e [term 1 LEADER]: Becoming Leader. State: Replica: bbe79c36f9de479bb753a665dcdbfd9e, State: Running, Role: LEADER
I20260620 18:43:52.978477 14174 consensus_queue.cc:237] T 00000000000000000000000000000000 P bbe79c36f9de479bb753a665dcdbfd9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe79c36f9de479bb753a665dcdbfd9e" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 38193 } } peers { permanent_uuid: "c230608d2fc847198dcce513367e58b7" member_type: VOTER last_known_addr { host: "127.23.115.61" port: 38393 } } peers { permanent_uuid: "a6400bf25e5745589c4cd0e78dc93f5a" member_type: VOTER last_known_addr { host: "127.23.115.60" port: 44645 } }
I20260620 18:43:52.982960 14182 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbe79c36f9de479bb753a665dcdbfd9e [sys.catalog]: SysCatalogTable state changed. Reason: New leader bbe79c36f9de479bb753a665dcdbfd9e. Latest consensus state: current_term: 1 leader_uuid: "bbe79c36f9de479bb753a665dcdbfd9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe79c36f9de479bb753a665dcdbfd9e" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 38193 } } peers { permanent_uuid: "c230608d2fc847198dcce513367e58b7" member_type: VOTER last_known_addr { host: "127.23.115.61" port: 38393 } } peers { permanent_uuid: "a6400bf25e5745589c4cd0e78dc93f5a" member_type: VOTER last_known_addr { host: "127.23.115.60" port: 44645 } } }
I20260620 18:43:52.983968 14182 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbe79c36f9de479bb753a665dcdbfd9e [sys.catalog]: This master's current role is: LEADER
I20260620 18:43:52.995414 14186 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:43:53.017310 14080 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.115.60" port: 44645 } attempt: 5
I20260620 18:43:53.027838 14186 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 18:43:53 dist-test-slave-d645 krb5kdc[13926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981032, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.61@KRBTEST.COM for kudu/127.23.115.60@KRBTEST.COM
W20260620 18:43:53.030808 14108 tablet.cc:2369] T 00000000000000000000000000000000 P a6400bf25e5745589c4cd0e78dc93f5a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:43:53.031107 13960 tablet.cc:2369] T 00000000000000000000000000000000 P bbe79c36f9de479bb753a665dcdbfd9e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:43:53.039827 14083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:53.017963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.60:44645 (local address 127.0.0.1:56290)
0620 18:43:53.018221 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:53.018243 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:53.019152 (+ 909us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:53.020202 (+ 1050us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:53.020210 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:53.021001 (+ 791us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:43:53.021275 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:53.021290 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:53.023564 (+ 2274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:53.023573 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:53.025015 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:53.025036 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:53.026394 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:53.026401 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:53.026756 (+ 355us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:53.027600 (+ 844us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:43:53.027645 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:43:53.032331 (+ 4686us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:43:53.036791 (+ 4460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:53.036797 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:53.036801 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:53.037367 (+ 566us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:53.037834 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:53.037839 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:53.037843 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:53.038008 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:53.038714 (+ 706us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:43:53.038719 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:43:53.038882 (+ 163us) client_negotiation.cc:769] Sending connection context
0620 18:43:53.039049 (+ 167us) client_negotiation.cc:240] Negotiation successful
0620 18:43:53.039361 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260620 18:43:53.041076 14157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:53.018324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56290 (local address 127.23.115.60:44645)
0620 18:43:53.018484 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:53.018489 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:53.018510 (+ 21us) server_negotiation.cc:407] Connection header received
0620 18:43:53.019391 (+ 881us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:53.019398 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:53.019495 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:53.019704 (+ 209us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:53.021746 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:53.023295 (+ 1549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:53.025256 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:53.025983 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:53.026107 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:53.032602 (+ 6495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:53.032609 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:53.032614 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:53.032657 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:53.036614 (+ 3957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:53.037525 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:53.037531 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:53.037535 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:53.037709 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:53.038125 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:53.038130 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:53.038134 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:53.038517 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:53.038654 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:53.040454 (+ 1800us) server_negotiation.cc:299] Negotiation successful
0620 18:43:53.040709 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 18:43:53.047339 14080 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c230608d2fc847198dcce513367e58b7: Bootstrap starting.
I20260620 18:43:53.051242 14080 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c230608d2fc847198dcce513367e58b7: Neither blocks nor log segments found. Creating new log.
I20260620 18:43:53.052588 14080 log.cc:824] T 00000000000000000000000000000000 P c230608d2fc847198dcce513367e58b7: Log is configured to *not* fsync() on all Append() calls
I20260620 18:43:53.059193 14129 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a6400bf25e5745589c4cd0e78dc93f5a [term 1 FOLLOWER]: Refusing update from remote peer bbe79c36f9de479bb753a665dcdbfd9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:43:53.060359 14182 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bbe79c36f9de479bb753a665dcdbfd9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6400bf25e5745589c4cd0e78dc93f5a" member_type: VOTER last_known_addr { host: "127.23.115.60" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:43:53.062470 14080 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c230608d2fc847198dcce513367e58b7: No bootstrap required, opened a new log
I20260620 18:43:53.073820 14080 raft_consensus.cc:348] T 00000000000000000000000000000000 P c230608d2fc847198dcce513367e58b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe79c36f9de479bb753a665dcdbfd9e" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 38193 } } peers { permanent_uuid: "c230608d2fc847198dcce513367e58b7" member_type: VOTER last_known_addr { host: "127.23.115.61" port: 38393 } } peers { permanent_uuid: "a6400bf25e5745589c4cd0e78dc93f5a" member_type: VOTER last_known_addr { host: "127.23.115.60" port: 44645 } }
I20260620 18:43:53.074358 14080 raft_consensus.cc:729] T 00000000000000000000000000000000 P c230608d2fc847198dcce513367e58b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c230608d2fc847198dcce513367e58b7, State: Initialized, Role: FOLLOWER
I20260620 18:43:53.074951 14080 consensus_queue.cc:260] T 00000000000000000000000000000000 P c230608d2fc847198dcce513367e58b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe79c36f9de479bb753a665dcdbfd9e" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 38193 } } peers { permanent_uuid: "c230608d2fc847198dcce513367e58b7" member_type: VOTER last_known_addr { host: "127.23.115.61" port: 38393 } } peers { permanent_uuid: "a6400bf25e5745589c4cd0e78dc93f5a" member_type: VOTER last_known_addr { host: "127.23.115.60" port: 44645 } }
I20260620 18:43:53.060967 13939 master_runner.cc:186] Error getting master registration for 127.23.115.61:38393: OK, instance_id { permanent_uuid: "c230608d2fc847198dcce513367e58b7" instance_seqno: 1781981032289159 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 18:43:53.082165 13939 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 18:43:53.084151 14080 sys_catalog.cc:564] T 00000000000000000000000000000000 P c230608d2fc847198dcce513367e58b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:43:53.084601 14054 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c230608d2fc847198dcce513367e58b7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:43:53.082103 14202 sys_catalog.cc:455] T 00000000000000000000000000000000 P c230608d2fc847198dcce513367e58b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe79c36f9de479bb753a665dcdbfd9e" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 38193 } } peers { permanent_uuid: "c230608d2fc847198dcce513367e58b7" member_type: VOTER last_known_addr { host: "127.23.115.61" port: 38393 } } peers { permanent_uuid: "a6400bf25e5745589c4cd0e78dc93f5a" member_type: VOTER last_known_addr { host: "127.23.115.60" port: 44645 } } }
I20260620 18:43:53.085645 14202 sys_catalog.cc:458] T 00000000000000000000000000000000 P c230608d2fc847198dcce513367e58b7 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:43:53.087143 14055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bbe79c36f9de479bb753a665dcdbfd9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c230608d2fc847198dcce513367e58b7"
I20260620 18:43:53.089027 14054 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c230608d2fc847198dcce513367e58b7 [term 1 FOLLOWER]: Refusing update from remote peer bbe79c36f9de479bb753a665dcdbfd9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:43:53.093612 14055 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c230608d2fc847198dcce513367e58b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbe79c36f9de479bb753a665dcdbfd9e in term 1.
I20260620 18:43:53.097205 14160 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6400bf25e5745589c4cd0e78dc93f5a [sys.catalog]: SysCatalogTable state changed. Reason: New leader bbe79c36f9de479bb753a665dcdbfd9e. Latest consensus state: current_term: 1 leader_uuid: "bbe79c36f9de479bb753a665dcdbfd9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe79c36f9de479bb753a665dcdbfd9e" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 38193 } } peers { permanent_uuid: "c230608d2fc847198dcce513367e58b7" member_type: VOTER last_known_addr { host: "127.23.115.61" port: 38393 } } peers { permanent_uuid: "a6400bf25e5745589c4cd0e78dc93f5a" member_type: VOTER last_known_addr { host: "127.23.115.60" port: 44645 } } }
I20260620 18:43:53.097524 14160 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6400bf25e5745589c4cd0e78dc93f5a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:43:53.100251 14174 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bbe79c36f9de479bb753a665dcdbfd9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c230608d2fc847198dcce513367e58b7" member_type: VOTER last_known_addr { host: "127.23.115.61" port: 38393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:43:53.101238 14160 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6400bf25e5745589c4cd0e78dc93f5a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bbe79c36f9de479bb753a665dcdbfd9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe79c36f9de479bb753a665dcdbfd9e" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 38193 } } peers { permanent_uuid: "c230608d2fc847198dcce513367e58b7" member_type: VOTER last_known_addr { host: "127.23.115.61" port: 38393 } } peers { permanent_uuid: "a6400bf25e5745589c4cd0e78dc93f5a" member_type: VOTER last_known_addr { host: "127.23.115.60" port: 44645 } } }
I20260620 18:43:53.101468 14160 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6400bf25e5745589c4cd0e78dc93f5a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:43:53.117241 14200 mvcc.cc:205] Tried to move back new op lower bound from 7298994311369900032 to 7298994311085010944. Current Snapshot: MvccSnapshot[applied={T|T < 7298994311369900032 or (T in {7298994311369900032})}]
I20260620 18:43:53.125929 14010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:53.014898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59814 (local address 127.23.115.62:38193)
0620 18:43:53.015102 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:53.015109 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:53.039264 (+ 24155us) server_negotiation.cc:407] Connection header received
0620 18:43:53.053526 (+ 14262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:53.053532 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:53.053621 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:53.053730 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:53.057839 (+ 4109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:53.059048 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:53.078532 (+ 19484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:53.079310 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:53.079422 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:53.097710 (+ 18288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:53.097718 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:53.097736 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:53.097786 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:53.111824 (+ 14038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:53.112778 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:53.112783 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:53.112787 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:53.112989 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:53.113451 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:53.113455 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:53.113458 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:53.113848 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:53.125020 (+ 11172us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:53.125229 (+ 209us) server_negotiation.cc:299] Negotiation successful
0620 18:43:53.125513 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 18:43:53.130286 14084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:53.056979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36508 (local address 127.23.115.61:38393)
0620 18:43:53.057172 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:53.057177 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:53.085132 (+ 27955us) server_negotiation.cc:407] Connection header received
0620 18:43:53.085381 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:53.085386 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:53.085467 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:53.085544 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:53.090205 (+ 4661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:53.091482 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:53.098788 (+ 7306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:53.099508 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:53.099634 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:53.114079 (+ 14445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:53.114088 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:53.114093 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:53.114140 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:53.118009 (+ 3869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:53.123060 (+ 5051us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:53.123067 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:53.123071 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:53.123278 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:53.126741 (+ 3463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:53.126748 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:53.126752 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:53.127230 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:53.127484 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:53.129583 (+ 2099us) server_negotiation.cc:299] Negotiation successful
0620 18:43:53.129862 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 18:43:53.134917 14194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:53.024411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56300 (local address 127.23.115.60:44645)
0620 18:43:53.041445 (+ 17034us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:53.041454 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:53.055915 (+ 14461us) server_negotiation.cc:407] Connection header received
0620 18:43:53.055980 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:53.055985 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:53.056125 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:53.056224 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:53.060314 (+ 4090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:53.073153 (+ 12839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:53.079009 (+ 5856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:53.079707 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:53.079848 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:53.095635 (+ 15787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:53.095642 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:53.095647 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:53.095691 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:53.108324 (+ 12633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:53.128248 (+ 19924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:53.128255 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:53.128259 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:53.128481 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:53.131021 (+ 2540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:53.131028 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:53.131032 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:53.131551 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:53.131814 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:53.133969 (+ 2155us) server_negotiation.cc:299] Negotiation successful
0620 18:43:53.134273 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16899,"thread_start_us":17034,"threads_started":1}
I20260620 18:43:53.141505 14174 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbe79c36f9de479bb753a665dcdbfd9e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bbe79c36f9de479bb753a665dcdbfd9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe79c36f9de479bb753a665dcdbfd9e" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 38193 } } peers { permanent_uuid: "c230608d2fc847198dcce513367e58b7" member_type: VOTER last_known_addr { host: "127.23.115.61" port: 38393 } } peers { permanent_uuid: "a6400bf25e5745589c4cd0e78dc93f5a" member_type: VOTER last_known_addr { host: "127.23.115.60" port: 44645 } } }
I20260620 18:43:53.141824 14174 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbe79c36f9de479bb753a665dcdbfd9e [sys.catalog]: This master's current role is: LEADER
I20260620 18:43:53.143812 14186 catalog_manager.cc:1347] Generated new cluster ID: e9b8b022690947ee81074f148d7e57c1
I20260620 18:43:53.144008 14186 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:43:53.158197 14202 sys_catalog.cc:455] T 00000000000000000000000000000000 P c230608d2fc847198dcce513367e58b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bbe79c36f9de479bb753a665dcdbfd9e. Latest consensus state: current_term: 1 leader_uuid: "bbe79c36f9de479bb753a665dcdbfd9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe79c36f9de479bb753a665dcdbfd9e" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 38193 } } peers { permanent_uuid: "c230608d2fc847198dcce513367e58b7" member_type: VOTER last_known_addr { host: "127.23.115.61" port: 38393 } } peers { permanent_uuid: "a6400bf25e5745589c4cd0e78dc93f5a" member_type: VOTER last_known_addr { host: "127.23.115.60" port: 44645 } } }
I20260620 18:43:53.158533 14202 sys_catalog.cc:458] T 00000000000000000000000000000000 P c230608d2fc847198dcce513367e58b7 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:43:53.158900 14212 sys_catalog.cc:455] T 00000000000000000000000000000000 P c230608d2fc847198dcce513367e58b7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bbe79c36f9de479bb753a665dcdbfd9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe79c36f9de479bb753a665dcdbfd9e" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 38193 } } peers { permanent_uuid: "c230608d2fc847198dcce513367e58b7" member_type: VOTER last_known_addr { host: "127.23.115.61" port: 38393 } } peers { permanent_uuid: "a6400bf25e5745589c4cd0e78dc93f5a" member_type: VOTER last_known_addr { host: "127.23.115.60" port: 44645 } } }
I20260620 18:43:53.159366 14212 sys_catalog.cc:458] T 00000000000000000000000000000000 P c230608d2fc847198dcce513367e58b7 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:43:53.171350 14182 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbe79c36f9de479bb753a665dcdbfd9e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bbe79c36f9de479bb753a665dcdbfd9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbe79c36f9de479bb753a665dcdbfd9e" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 38193 } } peers { permanent_uuid: "c230608d2fc847198dcce513367e58b7" member_type: VOTER last_known_addr { host: "127.23.115.61" port: 38393 } } peers { permanent_uuid: "a6400bf25e5745589c4cd0e78dc93f5a" member_type: VOTER last_known_addr { host: "127.23.115.60" port: 44645 } } }
I20260620 18:43:53.171700 14182 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbe79c36f9de479bb753a665dcdbfd9e [sys.catalog]: This master's current role is: LEADER
I20260620 18:43:53.177634 14186 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:43:53.180938 14186 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:43:53.215217 14186 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bbe79c36f9de479bb753a665dcdbfd9e: Generated new TSK 0
I20260620 18:43:53.216423 14186 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:43:53.251943 14220 catalog_manager.cc:1259] Loaded cluster ID: e9b8b022690947ee81074f148d7e57c1
I20260620 18:43:53.252132 14220 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c230608d2fc847198dcce513367e58b7: loading cluster ID for follower catalog manager: success
I20260620 18:43:53.255539 14220 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c230608d2fc847198dcce513367e58b7: acquiring CA information for follower catalog manager: success
I20260620 18:43:53.258178 14220 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c230608d2fc847198dcce513367e58b7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 18:43:53.381364 14010 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:43:53.360099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59830 (local address 127.23.115.62:38193)
0620 18:43:53.360329 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:53.360336 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:53.379443 (+ 19107us) server_negotiation.cc:407] Connection header received
0620 18:43:53.379764 (+ 321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:53.379770 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:53.379864 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:53.379948 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:53.381104 (+ 1156us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59830: BlockingRecv error: recv got EOF from 127.0.0.1:59830 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260620 18:43:53.383674 14194 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:43:53.368349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56316 (local address 127.23.115.60:44645)
0620 18:43:53.368545 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:53.368552 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:53.382121 (+ 13569us) server_negotiation.cc:407] Connection header received
0620 18:43:53.382349 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:53.382355 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:53.382430 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:53.382498 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:53.383437 (+ 939us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56316: BlockingRecv error: recv got EOF from 127.0.0.1:56316 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260620 18:43:53.385946 14084 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:43:53.383553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36518 (local address 127.23.115.61:38393)
0620 18:43:53.383779 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:53.383784 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:53.384356 (+ 572us) server_negotiation.cc:407] Connection header received
0620 18:43:53.384639 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:53.384649 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:53.384751 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:53.384819 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:53.385731 (+ 912us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36518: BlockingRecv error: recv got EOF from 127.0.0.1:36518 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
I20260620 18:43:53.437909 14084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:53.418108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36522 (local address 127.23.115.61:38393)
0620 18:43:53.418319 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:53.418326 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:53.430065 (+ 11739us) server_negotiation.cc:407] Connection header received
0620 18:43:53.430305 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:53.430311 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:53.430533 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:53.430715 (+ 182us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:43:53.431385 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:53.432541 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:53.434525 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:53.435241 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:53.435343 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:53.436359 (+ 1016us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:43:53.437148 (+ 789us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:43:53.437258 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:53.437478 (+ 220us) server_negotiation.cc:299] Negotiation successful
0620 18:43:53.437571 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
W20260620 18:43:53.478842 14194 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:43:53.471926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56330 (local address 127.23.115.60:44645)
0620 18:43:53.472117 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:53.472124 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:53.477162 (+ 5038us) server_negotiation.cc:407] Connection header received
0620 18:43:53.477410 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:53.477416 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:53.477546 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:53.477679 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:53.478572 (+ 893us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56330: BlockingRecv error: recv got EOF from 127.0.0.1:56330 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 18:43:53.490134 14084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:53.483486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36528 (local address 127.23.115.61:38393)
0620 18:43:53.483685 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:53.483692 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:53.484081 (+ 389us) server_negotiation.cc:407] Connection header received
0620 18:43:53.484341 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:53.484348 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:53.484456 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:53.484556 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:43:53.485006 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:53.486199 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:53.488119 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:53.488816 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:53.488948 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:53.489355 (+ 407us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:43:53.489532 (+ 177us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:43:53.489619 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:53.489747 (+ 128us) server_negotiation.cc:299] Negotiation successful
0620 18:43:53.489838 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260620 18:43:53.500890 14194 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:43:53.497202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56346 (local address 127.23.115.60:44645)
0620 18:43:53.497399 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:53.497405 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:53.499471 (+ 2066us) server_negotiation.cc:407] Connection header received
0620 18:43:53.499640 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:53.499646 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:53.499733 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:53.499845 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:53.500657 (+ 812us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56346: BlockingRecv error: recv got EOF from 127.0.0.1:56346 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 18:43:53.517381 14084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:53.506349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36542 (local address 127.23.115.61:38393)
0620 18:43:53.506524 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:53.506529 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:53.511505 (+ 4976us) server_negotiation.cc:407] Connection header received
0620 18:43:53.511683 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:53.511689 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:53.511773 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:53.511886 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:43:53.512301 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:53.513424 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:53.515302 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:53.515966 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:53.516085 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:53.516571 (+ 486us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:43:53.516716 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:43:53.516807 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:53.516966 (+ 159us) server_negotiation.cc:299] Negotiation successful
0620 18:43:53.517047 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260620 18:43:53.526595 14194 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:43:53.524212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56360 (local address 127.23.115.60:44645)
0620 18:43:53.524408 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:53.524414 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:53.525068 (+ 654us) server_negotiation.cc:407] Connection header received
0620 18:43:53.525275 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:53.525281 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:53.525389 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:53.525502 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:53.526356 (+ 854us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56360: BlockingRecv error: recv got EOF from 127.0.0.1:56360 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 18:43:53.541731 14084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:53.534573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36558 (local address 127.23.115.61:38393)
0620 18:43:53.534818 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:53.534824 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:53.535330 (+ 506us) server_negotiation.cc:407] Connection header received
0620 18:43:53.535524 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:53.535529 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:53.535614 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:53.535710 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:43:53.536383 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:53.537533 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:53.539477 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:53.540145 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:53.540267 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:53.540732 (+ 465us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:43:53.540872 (+ 140us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:43:53.540958 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:53.541109 (+ 151us) server_negotiation.cc:299] Negotiation successful
0620 18:43:53.541194 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
W20260620 18:43:53.555058 14194 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:43:53.552803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56376 (local address 127.23.115.60:44645)
0620 18:43:53.553049 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:53.553055 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:53.553604 (+ 549us) server_negotiation.cc:407] Connection header received
0620 18:43:53.553784 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:53.553798 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:53.553888 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:53.553997 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:53.554778 (+ 781us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56376: BlockingRecv error: recv got EOF from 127.0.0.1:56376 (error 108)
Metrics: {"server-negotiator.queue_time_us":127}
I20260620 18:43:53.573273 14084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:53.566146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36570 (local address 127.23.115.61:38393)
0620 18:43:53.566374 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:53.566379 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:53.566877 (+ 498us) server_negotiation.cc:407] Connection header received
0620 18:43:53.567100 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:53.567106 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:53.567188 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:53.567280 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:43:53.567667 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:53.568832 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:53.570828 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:53.571560 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:53.571698 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:53.572294 (+ 596us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:43:53.572500 (+ 206us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:43:53.572568 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:53.572839 (+ 271us) server_negotiation.cc:299] Negotiation successful
0620 18:43:53.572933 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260620 18:43:53.582415 14194 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:43:53.580216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56380 (local address 127.23.115.60:44645)
0620 18:43:53.580411 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:53.580417 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:53.580896 (+ 479us) server_negotiation.cc:407] Connection header received
0620 18:43:53.581096 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:53.581102 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:53.581196 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:53.581295 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:53.582196 (+ 901us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56380: BlockingRecv error: recv got EOF from 127.0.0.1:56380 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 18:43:53.608798 14084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:53.601855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36572 (local address 127.23.115.61:38393)
0620 18:43:53.602062 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:53.602068 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:53.602592 (+ 524us) server_negotiation.cc:407] Connection header received
0620 18:43:53.602816 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:53.602822 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:53.602913 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:53.603046 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:43:53.603561 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:53.604778 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:53.606675 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:53.607383 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:53.607502 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:53.608032 (+ 530us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:43:53.608161 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:43:53.608248 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:53.608427 (+ 179us) server_negotiation.cc:299] Negotiation successful
0620 18:43:53.608521 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260620 18:43:53.617578 14194 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:43:53.615312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56390 (local address 127.23.115.60:44645)
0620 18:43:53.615521 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:53.615527 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:53.616087 (+ 560us) server_negotiation.cc:407] Connection header received
0620 18:43:53.616249 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:53.616254 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:53.616360 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:53.616466 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:53.617350 (+ 884us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56390: BlockingRecv error: recv got EOF from 127.0.0.1:56390 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 18:43:53.658658 14084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:53.652480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36584 (local address 127.23.115.61:38393)
0620 18:43:53.652699 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:53.652704 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:53.653082 (+ 378us) server_negotiation.cc:407] Connection header received
0620 18:43:53.653303 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:53.653310 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:53.653405 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:53.653491 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:43:53.654021 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:53.654773 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:53.656685 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:53.657216 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:53.657328 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:53.657860 (+ 532us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:43:53.657972 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:43:53.658042 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:53.658245 (+ 203us) server_negotiation.cc:299] Negotiation successful
0620 18:43:53.658321 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260620 18:43:53.666131 14194 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:43:53.664253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56394 (local address 127.23.115.60:44645)
0620 18:43:53.664462 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:53.664468 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:53.664753 (+ 285us) server_negotiation.cc:407] Connection header received
0620 18:43:53.664910 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:53.664915 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:53.665014 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:53.665133 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:53.665881 (+ 748us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56394: BlockingRecv error: recv got EOF from 127.0.0.1:56394 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
I20260620 18:43:53.738394 14084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:53.733278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36600 (local address 127.23.115.61:38393)
0620 18:43:53.733474 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:53.733481 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:53.733791 (+ 310us) server_negotiation.cc:407] Connection header received
0620 18:43:53.733988 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:53.733992 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:53.734060 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:53.734126 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:43:53.734609 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:53.735401 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:53.736908 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:53.737290 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:53.737399 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:53.737787 (+ 388us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:43:53.737883 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:43:53.737947 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:53.738036 (+ 89us) server_negotiation.cc:299] Negotiation successful
0620 18:43:53.738097 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260620 18:43:53.746068 14194 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:43:53.744254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56398 (local address 127.23.115.60:44645)
0620 18:43:53.744407 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:53.744413 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:53.744799 (+ 386us) server_negotiation.cc:407] Connection header received
0620 18:43:53.744976 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:53.744980 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:53.745051 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:53.745133 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:53.745855 (+ 722us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56398: BlockingRecv error: recv got EOF from 127.0.0.1:56398 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 18:43:53.882248 14084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:53.877272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36604 (local address 127.23.115.61:38393)
0620 18:43:53.877494 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:53.877506 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:53.877776 (+ 270us) server_negotiation.cc:407] Connection header received
0620 18:43:53.877938 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:53.877942 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:53.878021 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:53.878100 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:43:53.878420 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:53.879180 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:53.880593 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:53.881011 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:53.881106 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:53.881514 (+ 408us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:43:53.881648 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:43:53.881743 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:53.881901 (+ 158us) server_negotiation.cc:299] Negotiation successful
0620 18:43:53.881981 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260620 18:43:53.891206 14194 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:43:53.888978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56408 (local address 127.23.115.60:44645)
0620 18:43:53.889198 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:53.889205 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:53.889642 (+ 437us) server_negotiation.cc:407] Connection header received
0620 18:43:53.889841 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:53.889845 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:53.889918 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:53.890002 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:53.890967 (+ 965us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56408: BlockingRecv error: recv got EOF from 127.0.0.1:56408 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 18:43:53.892678 14171 catalog_manager.cc:1259] Loaded cluster ID: e9b8b022690947ee81074f148d7e57c1
I20260620 18:43:53.892752 14171 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a6400bf25e5745589c4cd0e78dc93f5a: loading cluster ID for follower catalog manager: success
I20260620 18:43:53.895219 14171 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a6400bf25e5745589c4cd0e78dc93f5a: acquiring CA information for follower catalog manager: success
I20260620 18:43:53.896924 14171 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a6400bf25e5745589c4cd0e78dc93f5a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 18:43:54.155143 14084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:54.150308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36606 (local address 127.23.115.61:38393)
0620 18:43:54.150529 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:54.150535 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:54.150715 (+ 180us) server_negotiation.cc:407] Connection header received
0620 18:43:54.150970 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:54.150975 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:54.151062 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:54.151151 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:43:54.151550 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:54.152261 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:54.153555 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:54.153971 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:54.154076 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:54.154470 (+ 394us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:43:54.154572 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:43:54.154645 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:54.154801 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 18:43:54.154885 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 18:43:54.166992 14194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:54.161260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56416 (local address 127.23.115.60:44645)
0620 18:43:54.161466 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:54.161472 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:54.161591 (+ 119us) server_negotiation.cc:407] Connection header received
0620 18:43:54.161773 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:54.161776 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:54.161953 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:54.162054 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:43:54.162461 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:54.163214 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:54.164883 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:54.165291 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:54.165401 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:54.165867 (+ 466us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:43:54.166400 (+ 533us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:43:54.166535 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:54.166626 (+ 91us) server_negotiation.cc:299] Negotiation successful
0620 18:43:54.166711 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 18:43:54.180987 14194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:54.173485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56422 (local address 127.23.115.60:44645)
0620 18:43:54.173730 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:54.173737 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:54.175095 (+ 1358us) server_negotiation.cc:407] Connection header received
0620 18:43:54.175563 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:54.175568 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:54.175650 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:54.175711 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:43:54.176449 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:54.177390 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:54.178978 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:54.179622 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:54.179740 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:54.180225 (+ 485us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:43:54.180372 (+ 147us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:43:54.180460 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:54.180598 (+ 138us) server_negotiation.cc:299] Negotiation successful
0620 18:43:54.180681 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 18:43:54.181245 14084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:54.175179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36610 (local address 127.23.115.61:38393)
0620 18:43:54.175371 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:54.175378 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:54.175816 (+ 438us) server_negotiation.cc:407] Connection header received
0620 18:43:54.175976 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:54.175980 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:54.176044 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:54.176115 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:43:54.176726 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:54.177474 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:54.179409 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:54.179893 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:54.180012 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:54.180433 (+ 421us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:43:54.180569 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:43:54.180646 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:54.180837 (+ 191us) server_negotiation.cc:299] Negotiation successful
0620 18:43:54.180920 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 18:43:54.181559 14335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:54.173485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59838 (local address 127.23.115.62:38193)
0620 18:43:54.174047 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:54.174052 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:54.174437 (+ 385us) server_negotiation.cc:407] Connection header received
0620 18:43:54.174692 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:54.174699 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:54.174956 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:54.175126 (+ 170us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:43:54.175887 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:54.176841 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:54.179002 (+ 2161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:54.179620 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:54.179740 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:54.180380 (+ 640us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:43:54.180835 (+ 455us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:43:54.180936 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:54.181042 (+ 106us) server_negotiation.cc:299] Negotiation successful
0620 18:43:54.181145 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":348,"threads_started":1}
I20260620 18:43:54.187098 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 13939
I20260620 18:43:54.188127 14001 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:43:54.302106 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 13939
I20260620 18:43:54.319353 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 14013
I20260620 18:43:54.320444 14075 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:43:54.425714 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 14013
I20260620 18:43:54.442370 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 14087
I20260620 18:43:54.443459 14149 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:43:54.548187 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 14087
2026-06-20T18:43:54Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4461 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980676253068-24012-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:43:54 dist-test-slave-d645 krb5kdc[14344](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:43:54 dist-test-slave-d645 krb5kdc[14344](info): set up 2 sockets
Jun 20 18:43:54 dist-test-slave-d645 krb5kdc[14344](info): commencing operation
krb5kdc: starting...
W20260620 18:43:55.579504 24012 mini_kdc.cc:121] Time spent starting KDC: real 0.995s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980676253068-24012-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980676253068-24012-0/krb5kdc/test-user.keytab.
Jun 20 18:43:55 dist-test-slave-d645 krb5kdc[14344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981035, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:43:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:43:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.115.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.115.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.115.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab.
Entry for principal kudu/127.23.115.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab.
I20260620 18:43:55.830420 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980676253068-24012-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980676253068-24012-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980676253068-24012-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.115.62:34249
--webserver_interface=127.23.115.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab
--principal=kudu/127.23.115.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:46185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.115.62:34249,127.23.115.61:38737,127.23.115.60:33719
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:43:56.159099 14357 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 18:43:56.159425 14357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:43:56.159526 14357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:43:56.159631 14357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:43:56.167001 14357 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:43:56.167091 14357 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:43:56.167150 14357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:43:56.167215 14357 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:43:56.167291 14357 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:43:56.177212 14357 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:46185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980676253068-24012-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980676253068-24012-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.115.62:34249,127.23.115.61:38737,127.23.115.60:33719
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.62.keytab
--principal=kudu/127.23.115.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.115.62:34249
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.115.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980676253068-24012-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:43:56.178486 14357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:43:56.179777 14357 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:43:56.191260 14362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:56.191227 14365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:56.191494 14363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:56.192288 14357 server_base.cc:1034] running on GCE node
I20260620 18:43:56.193033 14357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:43:56.194391 14357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:43:56.195605 14357 hybrid_clock.cc:648] HybridClock initialized: now 1781981036195539 us; error 39 us; skew 500 ppm
I20260620 18:43:56.195924 14357 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:43:56 dist-test-slave-d645 krb5kdc[14344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981036, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:43:56.199131 14357 init.cc:377] Logged in from keytab as kudu/127.23.115.62@KRBTEST.COM (short username kudu)
I20260620 18:43:56.200632 14357 webserver.cc:460] Webserver started at http://127.23.115.62:32793/ using document root <none> and password file <none>
I20260620 18:43:56.201310 14357 fs_manager.cc:359] Metadata directory not provided
I20260620 18:43:56.201409 14357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:43:56.201713 14357 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:43:56.204715 14357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980676253068-24012-0/minicluster-data/master-0/data/instance:
uuid: "fcedba0908f545b481a6b7a65cd28ef6"
format_stamp: "Formatted at 2026-06-20 18:43:56 on dist-test-slave-d645"
server_key: "96a5b542d0afaf918ea963fdea771c55"
server_key_iv: "f88f00a8a6f8f56396506907f171e772"
server_key_version: "encryptionkey@0"
I20260620 18:43:56.205480 14357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980676253068-24012-0/minicluster-data/master-0/wal/instance:
uuid: "fcedba0908f545b481a6b7a65cd28ef6"
format_stamp: "Formatted at 2026-06-20 18:43:56 on dist-test-slave-d645"
server_key: "96a5b542d0afaf918ea963fdea771c55"
server_key_iv: "f88f00a8a6f8f56396506907f171e772"
server_key_version: "encryptionkey@0"
I20260620 18:43:56.210593 14357 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 18:43:56.214303 14372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:56.215624 14357 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 18:43:56.215786 14357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980676253068-24012-0/minicluster-data/master-0/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980676253068-24012-0/minicluster-data/master-0/wal
uuid: "fcedba0908f545b481a6b7a65cd28ef6"
format_stamp: "Formatted at 2026-06-20 18:43:56 on dist-test-slave-d645"
server_key: "96a5b542d0afaf918ea963fdea771c55"
server_key_iv: "f88f00a8a6f8f56396506907f171e772"
server_key_version: "encryptionkey@0"
I20260620 18:43:56.215991 14357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980676253068-24012-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980676253068-24012-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980676253068-24012-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:43:56.241030 14357 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:43:56.245267 14357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:43:56.245733 14357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:43:56.264531 14357 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.62:34249
I20260620 18:43:56.264541 14423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.62:34249 every 8 connection(s)
I20260620 18:43:56.265888 14357 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980676253068-24012-0/minicluster-data/master-0/data/info.pb
I20260620 18:43:56.269824 14424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:43:56.274072 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 14357
I20260620 18:43:56.274232 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980676253068-24012-0/minicluster-data/master-0/wal/instance
I20260620 18:43:56.274679 24012 external_mini_cluster.cc:1428] Setting key bc8f9f68fa8585bba48349d7c05d367f
I20260620 18:43:56.276829 14424 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.115.62" port: 34249 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:43:56 dist-test-slave-d645 krb5kdc[14344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981036, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.62@KRBTEST.COM for kudu/127.23.115.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.115.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.115.61@KRBTEST.COM" created.
I20260620 18:43:56.301549 14426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:56.280062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:34249 (local address 127.0.0.1:44834)
0620 18:43:56.281040 (+ 978us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:56.281091 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:56.283082 (+ 1991us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:56.283892 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:56.283899 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:56.284614 (+ 715us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:43:56.284940 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:56.284957 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:56.287152 (+ 2195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:56.287158 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:56.288671 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:56.288686 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:56.289874 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:56.289880 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:56.290034 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:56.290685 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:43:56.290736 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:43:56.293109 (+ 2373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:43:56.297255 (+ 4146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:56.297265 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:56.297276 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:56.297702 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:56.298205 (+ 503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:56.298208 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:56.298210 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:56.298367 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:56.299284 (+ 917us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:43:56.299291 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:43:56.299613 (+ 322us) client_negotiation.cc:769] Sending connection context
0620 18:43:56.299848 (+ 235us) client_negotiation.cc:240] Negotiation successful
0620 18:43:56.300302 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":718,"thread_start_us":506,"threads_started":1}
I20260620 18:43:56.301580 14427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:56.280062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44834 (local address 127.23.115.62:34249)
0620 18:43:56.281882 (+ 1820us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:56.281890 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:56.281940 (+ 50us) server_negotiation.cc:407] Connection header received
0620 18:43:56.283422 (+ 1482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:56.283430 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:56.283673 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:56.283863 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:56.285133 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:56.286952 (+ 1819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:56.288852 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:56.289616 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:56.289803 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:56.293374 (+ 3571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:56.293386 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:56.293404 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:56.293447 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:56.297092 (+ 3645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:56.297853 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:56.297862 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:56.297865 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:56.298067 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:56.298476 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:56.298482 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:56.298485 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:56.299107 (+ 622us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:56.299323 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:56.300040 (+ 717us) server_negotiation.cc:299] Negotiation successful
0620 18:43:56.300516 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":177,"server-negotiator.queue_time_us":1583,"spinlock_wait_cycles":15744,"thread_start_us":1374,"threads_started":1}
I20260620 18:43:56.305035 14424 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.115.61" port: 38737 } has no permanent_uuid. Determining permanent_uuid...
I20260620 18:43:56.306239 14426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:56.305742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.61:38737 (local address 127.0.0.1:40690)
0620 18:43:56.305949 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:56.306086 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.61:38737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260620 18:43:56.306505 14374 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.115.61:38737: connect: Connection refused (error 111)
I20260620 18:43:56.308187 14426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:56.307753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.61:38737 (local address 127.0.0.1:40704)
0620 18:43:56.307920 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:56.308020 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.61:38737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 18:43:56.308393 14424 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.115.61:38737: Network error: Client connection negotiation failed: client connection to 127.23.115.61:38737: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.115.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.61.keytab.
Entry for principal kudu/127.23.115.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.61.keytab.
I20260620 18:43:56.331310 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980676253068-24012-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980676253068-24012-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980676253068-24012-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980676253068-24012-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.115.61:38737
--webserver_interface=127.23.115.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.61.keytab
--principal=kudu/127.23.115.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:46185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.115.62:34249,127.23.115.61:38737,127.23.115.60:33719
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:43:56.357750 14424 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.115.61" port: 38737 } attempt: 1
I20260620 18:43:56.359071 14426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:56.358563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.61:38737 (local address 127.0.0.1:40714)
0620 18:43:56.358776 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:56.358890 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.61:38737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260620 18:43:56.360061 14426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:56.359687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.61:38737 (local address 127.0.0.1:40724)
0620 18:43:56.359843 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:56.359926 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.61:38737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260620 18:43:56.360277 14424 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.115.61:38737: Network error: Client connection negotiation failed: client connection to 127.23.115.61:38737: connect: Connection refused (error 111)
I20260620 18:43:56.428539 14424 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.115.61" port: 38737 } attempt: 2
I20260620 18:43:56.429680 14426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:56.429214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.61:38737 (local address 127.0.0.1:40740)
0620 18:43:56.429453 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:56.429540 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.61:38737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260620 18:43:56.430620 14426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:56.430261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.61:38737 (local address 127.0.0.1:40744)
0620 18:43:56.430408 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:56.430490 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.61:38737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 18:43:56.430788 14424 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.115.61:38737: Network error: Client connection negotiation failed: client connection to 127.23.115.61:38737: connect: Connection refused (error 111)
I20260620 18:43:56.522022 14424 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.115.61" port: 38737 } attempt: 3
I20260620 18:43:56.523118 14426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:56.522643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.61:38737 (local address 127.0.0.1:40746)
0620 18:43:56.522860 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:56.522981 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.61:38737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260620 18:43:56.524025 14426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:56.523692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.61:38737 (local address 127.0.0.1:40758)
0620 18:43:56.523832 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:56.523898 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.61:38737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260620 18:43:56.524187 14424 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.115.61:38737: Network error: Client connection negotiation failed: client connection to 127.23.115.61:38737: connect: Connection refused (error 111)
W20260620 18:43:56.647601 14431 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 18:43:56.647958 14431 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:43:56.648053 14431 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:43:56.648168 14431 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:43:56.655436 14431 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:43:56.655539 14431 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:43:56.655625 14431 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:43:56.655689 14431 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:43:56.655776 14431 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:43:56.667419 14424 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.115.61" port: 38737 } attempt: 4
I20260620 18:43:56.667600 14431 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:46185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980676253068-24012-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980676253068-24012-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.23.115.62:34249,127.23.115.61:38737,127.23.115.60:33719
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.61.keytab
--principal=kudu/127.23.115.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.115.61:38737
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980676253068-24012-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.23.115.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980676253068-24012-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:43:56.668655 14426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:56.668108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.61:38737 (local address 127.0.0.1:40762)
0620 18:43:56.668398 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:56.668521 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.61:38737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163}
I20260620 18:43:56.669191 14431 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:43:56.669812 14426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:56.669346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.61:38737 (local address 127.0.0.1:40766)
0620 18:43:56.669549 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:56.669675 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.61:38737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260620 18:43:56.670009 14424 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.115.61:38737: Network error: Client connection negotiation failed: client connection to 127.23.115.61:38737: connect: Connection refused (error 111)
I20260620 18:43:56.670634 14431 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:43:56.681761 14436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:56.681734 14439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:56.681893 14437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:56.683422 14431 server_base.cc:1034] running on GCE node
I20260620 18:43:56.684276 14431 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:43:56.685685 14431 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:43:56.686920 14431 hybrid_clock.cc:648] HybridClock initialized: now 1781981036686828 us; error 66 us; skew 500 ppm
I20260620 18:43:56.687273 14431 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:43:56 dist-test-slave-d645 krb5kdc[14344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981036, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:43:56.690434 14431 init.cc:377] Logged in from keytab as kudu/127.23.115.61@KRBTEST.COM (short username kudu)
I20260620 18:43:56.691900 14431 webserver.cc:460] Webserver started at http://127.23.115.61:41975/ using document root <none> and password file <none>
I20260620 18:43:56.692553 14431 fs_manager.cc:359] Metadata directory not provided
I20260620 18:43:56.692657 14431 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:43:56.692947 14431 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:43:56.695883 14431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980676253068-24012-0/minicluster-data/master-1/data/instance:
uuid: "cabe4f44108242658f56c8cc76a22f31"
format_stamp: "Formatted at 2026-06-20 18:43:56 on dist-test-slave-d645"
server_key: "a51829693ef0ea328a83b47290d47421"
server_key_iv: "e62697c3ae2f9991e43871ed5ff48905"
server_key_version: "encryptionkey@0"
I20260620 18:43:56.696656 14431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980676253068-24012-0/minicluster-data/master-1/wal/instance:
uuid: "cabe4f44108242658f56c8cc76a22f31"
format_stamp: "Formatted at 2026-06-20 18:43:56 on dist-test-slave-d645"
server_key: "a51829693ef0ea328a83b47290d47421"
server_key_iv: "e62697c3ae2f9991e43871ed5ff48905"
server_key_version: "encryptionkey@0"
I20260620 18:43:56.701617 14431 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:43:56.707695 14446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:56.709511 14431 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.003s
I20260620 18:43:56.709802 14431 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980676253068-24012-0/minicluster-data/master-1/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980676253068-24012-0/minicluster-data/master-1/wal
uuid: "cabe4f44108242658f56c8cc76a22f31"
format_stamp: "Formatted at 2026-06-20 18:43:56 on dist-test-slave-d645"
server_key: "a51829693ef0ea328a83b47290d47421"
server_key_iv: "e62697c3ae2f9991e43871ed5ff48905"
server_key_version: "encryptionkey@0"
I20260620 18:43:56.710168 14431 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980676253068-24012-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980676253068-24012-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980676253068-24012-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:43:56.733539 14431 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:43:56.737985 14431 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:43:56.738246 14431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:43:56.756588 14431 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.61:38737
I20260620 18:43:56.756608 14497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.61:38737 every 8 connection(s)
I20260620 18:43:56.757900 14431 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980676253068-24012-0/minicluster-data/master-1/data/info.pb
I20260620 18:43:56.761688 14498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:43:56.765044 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 14431
I20260620 18:43:56.765213 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980676253068-24012-0/minicluster-data/master-1/wal/instance
I20260620 18:43:56.765625 24012 external_mini_cluster.cc:1428] Setting key 8f32034314dac018a0a99e58bafe5e0b
I20260620 18:43:56.768700 14498 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.115.62" port: 34249 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:43:56 dist-test-slave-d645 krb5kdc[14344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981036, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.61@KRBTEST.COM for kudu/127.23.115.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.115.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.115.60@KRBTEST.COM" created.
I20260620 18:43:56.790320 14427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:56.771361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44850 (local address 127.23.115.62:34249)
0620 18:43:56.771577 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:56.771583 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:56.772851 (+ 1268us) server_negotiation.cc:407] Connection header received
0620 18:43:56.774814 (+ 1963us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:56.774820 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:56.774906 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:56.775047 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:56.776520 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:56.777424 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:56.778959 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:56.779518 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:56.779640 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:56.783326 (+ 3686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:56.783334 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:56.783338 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:56.783470 (+ 132us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:56.787053 (+ 3583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:56.787831 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:56.787838 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:56.787841 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:56.787963 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:56.788442 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:56.788448 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:56.788452 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:56.788801 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:56.788946 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:56.789703 (+ 757us) server_negotiation.cc:299] Negotiation successful
0620 18:43:56.789993 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 18:43:56.791437 14500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:56.771844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:34249 (local address 127.0.0.1:44850)
0620 18:43:56.772673 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:56.772721 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:56.774510 (+ 1789us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:56.775253 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:56.775262 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:56.775784 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:43:56.776338 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:56.776356 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:56.777607 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:56.777611 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:56.778809 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:56.778821 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:56.779748 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:56.779754 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:56.779956 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:56.780642 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:43:56.780665 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:43:56.783079 (+ 2414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:43:56.787198 (+ 4119us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:56.787207 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:56.787220 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:56.787679 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:56.788074 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:56.788078 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:56.788080 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:56.788258 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:56.788946 (+ 688us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:43:56.788955 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:43:56.789316 (+ 361us) client_negotiation.cc:769] Sending connection context
0620 18:43:56.789596 (+ 280us) client_negotiation.cc:240] Negotiation successful
0620 18:43:56.790105 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":342,"threads_started":1}
I20260620 18:43:56.793249 14498 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.115.61" port: 38737 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:43:56 dist-test-slave-d645 krb5kdc[14344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981036, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.61@KRBTEST.COM for kudu/127.23.115.61@KRBTEST.COM
I20260620 18:43:56.808090 14500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:56.793998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.61:38737 (local address 127.0.0.1:40774)
0620 18:43:56.794213 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:56.794234 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:56.794335 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:56.795285 (+ 950us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:56.795290 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:56.795727 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:43:56.795900 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:56.795910 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:56.797302 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:56.797308 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:56.798468 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:56.798480 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:56.799628 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:56.799633 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:56.799770 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:56.800378 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:43:56.800396 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:43:56.802517 (+ 2121us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:43:56.805518 (+ 3001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:56.805523 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:56.805526 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:56.805958 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:56.806332 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:56.806336 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:56.806339 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:56.806489 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:56.807159 (+ 670us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:43:56.807164 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:43:56.807307 (+ 143us) client_negotiation.cc:769] Sending connection context
0620 18:43:56.807446 (+ 139us) client_negotiation.cc:240] Negotiation successful
0620 18:43:56.807710 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 18:43:56.808111 14502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:56.794181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40774 (local address 127.23.115.61:38737)
0620 18:43:56.794849 (+ 668us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:56.794855 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:56.794880 (+ 25us) server_negotiation.cc:407] Connection header received
0620 18:43:56.794921 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:56.794953 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:56.795140 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:56.795250 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:56.796072 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:56.797169 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:56.798617 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:56.799459 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:56.799578 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:56.802744 (+ 3166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:56.802756 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:56.802770 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:56.802802 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:56.805398 (+ 2596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:56.806081 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:56.806088 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:56.806090 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:56.806213 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:56.806617 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:56.806622 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:56.806624 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:56.807022 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:56.807196 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:56.807618 (+ 422us) server_negotiation.cc:299] Negotiation successful
0620 18:43:56.807813 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":501,"thread_start_us":364,"threads_started":1}
I20260620 18:43:56.810544 14498 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.115.60" port: 33719 } has no permanent_uuid. Determining permanent_uuid...
I20260620 18:43:56.811666 14500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:56.811166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.60:33719 (local address 127.0.0.1:32882)
0620 18:43:56.811352 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:56.811486 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.60:33719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260620 18:43:56.811902 14448 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.115.60:33719: connect: Connection refused (error 111)
I20260620 18:43:56.813373 14500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:56.813041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.60:33719 (local address 127.0.0.1:32892)
0620 18:43:56.813198 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:56.813277 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.60:33719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 18:43:56.813584 14498 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.115.60:33719: Network error: Client connection negotiation failed: client connection to 127.23.115.60:33719: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.115.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.60.keytab.
Entry for principal kudu/127.23.115.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.60.keytab.
I20260620 18:43:56.820438 24012 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6x5P9o/build/asan/bin/kudu
/tmp/dist-test-task6x5P9o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980676253068-24012-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980676253068-24012-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980676253068-24012-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980676253068-24012-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.115.60:33719
--webserver_interface=127.23.115.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.60.keytab
--principal=kudu/127.23.115.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.115.20:46185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.115.62:34249,127.23.115.61:38737,127.23.115.60:33719
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980676253068-24012-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980676253068-24012-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980676253068-24012-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:43:56.862952 14498 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.115.60" port: 33719 } attempt: 1
I20260620 18:43:56.864234 14500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:56.863697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.60:33719 (local address 127.0.0.1:32894)
0620 18:43:56.863920 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:56.864043 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.60:33719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260620 18:43:56.865343 14500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:56.864959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.60:33719 (local address 127.0.0.1:32906)
0620 18:43:56.865124 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:56.865214 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.60:33719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 18:43:56.865600 14498 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.115.60:33719: Network error: Client connection negotiation failed: client connection to 127.23.115.60:33719: connect: Connection refused (error 111)
I20260620 18:43:56.933846 14498 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.115.60" port: 33719 } attempt: 2
I20260620 18:43:56.935076 14500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:56.934519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.60:33719 (local address 127.0.0.1:32912)
0620 18:43:56.934738 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:56.934850 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.60:33719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260620 18:43:56.936211 14500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:56.935831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.60:33719 (local address 127.0.0.1:32926)
0620 18:43:56.935982 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:56.936069 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.60:33719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 18:43:56.936460 14498 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.115.60:33719: Network error: Client connection negotiation failed: client connection to 127.23.115.60:33719: connect: Connection refused (error 111)
I20260620 18:43:56.969269 14424 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.115.61" port: 38737 } attempt: 5
Jun 20 18:43:56 dist-test-slave-d645 krb5kdc[14344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981036, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.62@KRBTEST.COM for kudu/127.23.115.61@KRBTEST.COM
I20260620 18:43:56.982532 14502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:56.970104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40776 (local address 127.23.115.61:38737)
0620 18:43:56.970310 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:56.970315 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:56.970336 (+ 21us) server_negotiation.cc:407] Connection header received
0620 18:43:56.970479 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:56.970485 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:56.970567 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:56.970650 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:56.971491 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:56.972203 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:56.973333 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:56.973787 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:56.973913 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:56.976888 (+ 2975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:56.976893 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:56.976896 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:56.976928 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:56.979843 (+ 2915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:56.980541 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:56.980547 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:56.980550 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:56.980697 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:56.981082 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:56.981088 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:56.981091 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:56.981397 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:56.981544 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:56.981950 (+ 406us) server_negotiation.cc:299] Negotiation successful
0620 18:43:56.982180 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260620 18:43:56.982537 14426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:56.970001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.61:38737 (local address 127.0.0.1:40776)
0620 18:43:56.970198 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:56.970219 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:56.970352 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:56.970707 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:56.970713 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:56.971143 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:43:56.971341 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:56.971351 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:56.972335 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:56.972340 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:56.973185 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:56.973198 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:56.973933 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:56.973939 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:56.974135 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:56.974716 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:43:56.974739 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:43:56.976700 (+ 1961us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:43:56.979992 (+ 3292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:56.979998 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:56.980003 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:56.980414 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:56.980815 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:56.980821 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:56.980825 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:56.980971 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:56.981520 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:43:56.981525 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:43:56.981691 (+ 166us) client_negotiation.cc:769] Sending connection context
0620 18:43:56.981835 (+ 144us) client_negotiation.cc:240] Negotiation successful
0620 18:43:56.982141 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 18:43:56.983723 14424 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.115.60" port: 33719 } has no permanent_uuid. Determining permanent_uuid...
I20260620 18:43:56.984711 14426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:56.984326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.60:33719 (local address 127.0.0.1:32930)
0620 18:43:56.984489 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:56.984568 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.60:33719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 18:43:56.985616 14426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:56.985285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.60:33719 (local address 127.0.0.1:32946)
0620 18:43:56.985426 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:56.985516 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.60:33719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 18:43:56.985816 14424 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.115.60:33719: Network error: Client connection negotiation failed: client connection to 127.23.115.60:33719: connect: Connection refused (error 111)
I20260620 18:43:57.027693 14498 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.115.60" port: 33719 } attempt: 3
I20260620 18:43:57.028911 14500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:57.028364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.60:33719 (local address 127.0.0.1:32952)
0620 18:43:57.028596 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:57.028711 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.60:33719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260620 18:43:57.030066 14500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:57.029619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.60:33719 (local address 127.0.0.1:32968)
0620 18:43:57.029864 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:57.029943 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.60:33719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146}
W20260620 18:43:57.030303 14498 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.115.60:33719: Network error: Client connection negotiation failed: client connection to 127.23.115.60:33719: connect: Connection refused (error 111)
I20260620 18:43:57.037050 14424 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.115.60" port: 33719 } attempt: 1
I20260620 18:43:57.038126 14426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:57.037692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.60:33719 (local address 127.0.0.1:32972)
0620 18:43:57.037875 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:57.037963 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.60:33719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 18:43:57.039227 14426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:57.038839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.60:33719 (local address 127.0.0.1:32978)
0620 18:43:57.039030 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:57.039107 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.60:33719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260620 18:43:57.039436 14424 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.115.60:33719: Network error: Client connection negotiation failed: client connection to 127.23.115.60:33719: connect: Connection refused (error 111)
I20260620 18:43:57.107692 14424 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.115.60" port: 33719 } attempt: 2
I20260620 18:43:57.108790 14426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:57.108319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.60:33719 (local address 127.0.0.1:32994)
0620 18:43:57.108532 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:57.108618 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.60:33719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260620 18:43:57.109920 14426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:57.109526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.60:33719 (local address 127.0.0.1:33008)
0620 18:43:57.109708 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:57.109813 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.60:33719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260620 18:43:57.110131 14424 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.115.60:33719: Network error: Client connection negotiation failed: client connection to 127.23.115.60:33719: connect: Connection refused (error 111)
W20260620 18:43:57.139886 14505 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 18:43:57.140195 14505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:43:57.140302 14505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:43:57.140405 14505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:43:57.147735 14505 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:43:57.147841 14505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:43:57.147936 14505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:43:57.148006 14505 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:43:57.148084 14505 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:43:57.158023 14505 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.115.20:46185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980676253068-24012-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980676253068-24012-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.23.115.62:34249,127.23.115.61:38737,127.23.115.60:33719
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980676253068-24012-0/krb5kdc/kudu_127.23.115.60.keytab
--principal=kudu/127.23.115.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.115.60:33719
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980676253068-24012-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.23.115.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980676253068-24012-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:43:57.159343 14505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:43:57.160578 14505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:43:57.172062 14510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:57.172026 14513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:43:57.172096 14511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:43:57.173041 14505 server_base.cc:1034] running on GCE node
I20260620 18:43:57.173486 14498 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.115.60" port: 33719 } attempt: 4
I20260620 18:43:57.173818 14505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:43:57.174463 14500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:57.174030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.60:33719 (local address 127.0.0.1:33012)
0620 18:43:57.174234 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:57.174315 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.60:33719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 18:43:57.175289 14505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:43:57.175510 14500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:57.175118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.60:33719 (local address 127.0.0.1:33022)
0620 18:43:57.175304 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:57.175394 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.60:33719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260620 18:43:57.175740 14498 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.115.60:33719: Network error: Client connection negotiation failed: client connection to 127.23.115.60:33719: connect: Connection refused (error 111)
I20260620 18:43:57.176517 14505 hybrid_clock.cc:648] HybridClock initialized: now 1781981037176443 us; error 49 us; skew 500 ppm
I20260620 18:43:57.176822 14505 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:43:57 dist-test-slave-d645 krb5kdc[14344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981037, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:43:57.179950 14505 init.cc:377] Logged in from keytab as kudu/127.23.115.60@KRBTEST.COM (short username kudu)
I20260620 18:43:57.181300 14505 webserver.cc:460] Webserver started at http://127.23.115.60:41187/ using document root <none> and password file <none>
I20260620 18:43:57.181922 14505 fs_manager.cc:359] Metadata directory not provided
I20260620 18:43:57.182005 14505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:43:57.182282 14505 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:43:57.185343 14505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980676253068-24012-0/minicluster-data/master-2/data/instance:
uuid: "2f68ca29c2dd4016b243df8733326ee9"
format_stamp: "Formatted at 2026-06-20 18:43:57 on dist-test-slave-d645"
server_key: "cfac389510a1f290c2dba7b9614ac838"
server_key_iv: "559a99ad2045b058210a9874f59ab079"
server_key_version: "encryptionkey@0"
I20260620 18:43:57.186079 14505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980676253068-24012-0/minicluster-data/master-2/wal/instance:
uuid: "2f68ca29c2dd4016b243df8733326ee9"
format_stamp: "Formatted at 2026-06-20 18:43:57 on dist-test-slave-d645"
server_key: "cfac389510a1f290c2dba7b9614ac838"
server_key_iv: "559a99ad2045b058210a9874f59ab079"
server_key_version: "encryptionkey@0"
I20260620 18:43:57.191144 14505 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260620 18:43:57.194911 14520 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:43:57.196424 14505 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:43:57.196592 14505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980676253068-24012-0/minicluster-data/master-2/data,/tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980676253068-24012-0/minicluster-data/master-2/wal
uuid: "2f68ca29c2dd4016b243df8733326ee9"
format_stamp: "Formatted at 2026-06-20 18:43:57 on dist-test-slave-d645"
server_key: "cfac389510a1f290c2dba7b9614ac838"
server_key_iv: "559a99ad2045b058210a9874f59ab079"
server_key_version: "encryptionkey@0"
I20260620 18:43:57.196803 14505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980676253068-24012-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980676253068-24012-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980676253068-24012-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:43:57.216339 14424 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.115.60" port: 33719 } attempt: 3
I20260620 18:43:57.217442 14426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:57.216965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.60:33719 (local address 127.0.0.1:33034)
0620 18:43:57.217198 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:57.217285 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.60:33719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260620 18:43:57.218429 14426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:57.218096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.60:33719 (local address 127.0.0.1:33050)
0620 18:43:57.218255 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:57.218326 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.115.60:33719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 18:43:57.218611 14424 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.115.60:33719: Network error: Client connection negotiation failed: client connection to 127.23.115.60:33719: connect: Connection refused (error 111)
I20260620 18:43:57.234131 14505 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:43:57.238394 14505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:43:57.238649 14505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:43:57.256920 14505 rpc_server.cc:307] RPC server started. Bound to: 127.23.115.60:33719
I20260620 18:43:57.256943 14571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.115.60:33719 every 8 connection(s)
I20260620 18:43:57.258260 14505 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980676253068-24012-0/minicluster-data/master-2/data/info.pb
I20260620 18:43:57.261885 14572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:43:57.264235 24012 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6x5P9o/build/asan/bin/kudu as pid 14505
I20260620 18:43:57.264421 24012 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6x5P9o/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980676253068-24012-0/minicluster-data/master-2/wal/instance
I20260620 18:43:57.264925 24012 external_mini_cluster.cc:1428] Setting key e58612bf3a8bd8bae8f18d934b60e212
I20260620 18:43:57.273080 14572 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.115.62" port: 34249 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:43:57 dist-test-slave-d645 krb5kdc[14344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981035, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.115.60@KRBTEST.COM
Jun 20 18:43:57 dist-test-slave-d645 krb5kdc[14344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981037, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.60@KRBTEST.COM for kudu/127.23.115.62@KRBTEST.COM
I20260620 18:43:57.289841 14575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:57.266728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33062 (local address 127.23.115.60:33719)
0620 18:43:57.267615 (+ 887us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:57.267632 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:57.267685 (+ 53us) server_negotiation.cc:407] Connection header received
0620 18:43:57.269877 (+ 2192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:57.269907 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:57.270430 (+ 523us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:57.270884 (+ 454us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:57.272553 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:57.274263 (+ 1710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:57.275935 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:57.277012 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:57.277191 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:57.281482 (+ 4291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:57.281490 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:57.281502 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:57.281540 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:57.285259 (+ 3719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:57.285957 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:57.285965 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:57.285972 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:57.286126 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:57.286514 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:57.286517 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:57.286520 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:57.287102 (+ 582us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:57.287391 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:57.288030 (+ 639us) server_negotiation.cc:299] Negotiation successful
0620 18:43:57.288479 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":637,"thread_start_us":366,"threads_started":1}
I20260620 18:43:57.290750 14427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:57.276103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44862 (local address 127.23.115.62:34249)
0620 18:43:57.276351 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:57.276358 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:57.277003 (+ 645us) server_negotiation.cc:407] Connection header received
0620 18:43:57.277191 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:57.277198 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:57.277280 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:57.277365 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:57.278707 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:57.279450 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:57.280717 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:57.281142 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:57.281250 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:57.285107 (+ 3857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:57.285113 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:57.285116 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:57.285152 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:57.287624 (+ 2472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:57.288350 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:57.288357 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:57.288360 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:57.288525 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:57.288923 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:57.288929 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:57.288933 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:57.289276 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:57.289438 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:57.290135 (+ 697us) server_negotiation.cc:299] Negotiation successful
0620 18:43:57.290419 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 18:43:57.291776 14576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:57.276019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.62:34249 (local address 127.0.0.1:44862)
0620 18:43:57.276827 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:57.276872 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:57.277030 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:57.277571 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:57.277575 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:57.278313 (+ 738us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:43:57.278563 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:57.278572 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:57.279590 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:57.279594 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:57.280545 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:57.280559 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:57.281364 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:57.281373 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:57.281523 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:57.282124 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:43:57.282146 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:43:57.284906 (+ 2760us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:43:57.287746 (+ 2840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:57.287757 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:57.287765 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:57.288164 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:57.288665 (+ 501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:57.288668 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:57.288671 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:57.288809 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:57.289480 (+ 671us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:43:57.289489 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:43:57.289734 (+ 245us) client_negotiation.cc:769] Sending connection context
0620 18:43:57.291012 (+ 1278us) client_negotiation.cc:240] Negotiation successful
0620 18:43:57.291321 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":347,"threads_started":1}
I20260620 18:43:57.293262 14572 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.115.61" port: 38737 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:43:57 dist-test-slave-d645 krb5kdc[14344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981037, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.60@KRBTEST.COM for kudu/127.23.115.61@KRBTEST.COM
Jun 20 18:43:57 dist-test-slave-d645 krb5kdc[14344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981035, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.115.62@KRBTEST.COM
I20260620 18:43:57.310925 14502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:57.294219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40778 (local address 127.23.115.61:38737)
0620 18:43:57.294414 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:57.294420 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:57.294439 (+ 19us) server_negotiation.cc:407] Connection header received
0620 18:43:57.294628 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:57.294633 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:57.294713 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:57.294797 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:57.295809 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:57.296608 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:57.298691 (+ 2083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:57.299306 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:57.299509 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:57.304160 (+ 4651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:57.304170 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:57.304175 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:57.304243 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:57.307827 (+ 3584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:57.308571 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:57.308576 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:57.308578 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:57.308841 (+ 263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:57.309273 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:57.309279 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:57.309283 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:57.309618 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:57.309799 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:57.310286 (+ 487us) server_negotiation.cc:299] Negotiation successful
0620 18:43:57.310556 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 18:43:57.310912 14576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:57.294198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.61:38737 (local address 127.0.0.1:40778)
0620 18:43:57.294372 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:57.294390 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:57.294500 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:57.295001 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:57.295006 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:57.295458 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:43:57.295632 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:57.295644 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:57.296867 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:57.296873 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:57.298363 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:57.298392 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:57.299517 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:57.299522 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:57.299772 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:57.300349 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:43:57.300368 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:43:57.303837 (+ 3469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:43:57.308001 (+ 4164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:57.308008 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:57.308013 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:57.308449 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:57.308988 (+ 539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:57.308992 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:57.308995 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:57.309150 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:57.309783 (+ 633us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:43:57.309787 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:43:57.309975 (+ 188us) client_negotiation.cc:769] Sending connection context
0620 18:43:57.310148 (+ 173us) client_negotiation.cc:240] Negotiation successful
0620 18:43:57.310480 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260620 18:43:57.311808 14427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:57.294658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44866 (local address 127.23.115.62:34249)
0620 18:43:57.294806 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:57.294810 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:57.294823 (+ 13us) server_negotiation.cc:407] Connection header received
0620 18:43:57.294864 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:57.294867 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:57.294919 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:57.295029 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:57.296347 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:57.297237 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:57.299454 (+ 2217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:57.300080 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:57.300210 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:57.305748 (+ 5538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:57.305755 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:57.305759 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:57.305805 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:57.308545 (+ 2740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:57.309355 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:57.309360 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:57.309362 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:57.309501 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:57.309922 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:57.309929 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:57.309932 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:57.310327 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:57.310610 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:57.311255 (+ 645us) server_negotiation.cc:299] Negotiation successful
0620 18:43:57.311507 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 18:43:57.312978 14572 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.115.60" port: 33719 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:43:57 dist-test-slave-d645 krb5kdc[14344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981037, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.60@KRBTEST.COM for kudu/127.23.115.60@KRBTEST.COM
Jun 20 18:43:57 dist-test-slave-d645 krb5kdc[14344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981035, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.115.61@KRBTEST.COM
I20260620 18:43:57.326089 14576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:57.313755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.60:33719 (local address 127.0.0.1:33074)
0620 18:43:57.314198 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:57.314219 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:57.314437 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:57.314855 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:57.314858 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:57.315240 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:43:57.315375 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:57.315383 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:57.316291 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:57.316296 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:57.317127 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:57.317137 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:57.317760 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:57.317766 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:57.317943 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:57.318461 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:43:57.318478 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:43:57.320786 (+ 2308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:43:57.323690 (+ 2904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:57.323695 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:57.323700 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:57.324177 (+ 477us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:57.324494 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:57.324499 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:57.324503 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:57.324629 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:57.325120 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:43:57.325125 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:43:57.325258 (+ 133us) client_negotiation.cc:769] Sending connection context
0620 18:43:57.325378 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 18:43:57.325669 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120}
I20260620 18:43:57.326133 14575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:57.313942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33074 (local address 127.23.115.60:33719)
0620 18:43:57.314111 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:57.314115 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:57.314338 (+ 223us) server_negotiation.cc:407] Connection header received
0620 18:43:57.314557 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:57.314561 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:57.314626 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:57.314685 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:57.315497 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:57.316176 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:57.317246 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:57.317637 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:57.317739 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:57.320974 (+ 3235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:57.320980 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:57.320983 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:57.321010 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:57.323556 (+ 2546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:57.324284 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:57.324288 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:57.324291 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:57.324398 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:57.324721 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:57.324726 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:57.324729 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:57.325016 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:57.325138 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:57.325470 (+ 332us) server_negotiation.cc:299] Negotiation successful
0620 18:43:57.325841 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":145,"server-negotiator.queue_time_us":51}
I20260620 18:43:57.328135 14502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:57.313525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40788 (local address 127.23.115.61:38737)
0620 18:43:57.313691 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:57.313697 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:57.313715 (+ 18us) server_negotiation.cc:407] Connection header received
0620 18:43:57.314196 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:57.314203 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:57.314400 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:57.314522 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:57.315926 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:57.316665 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:57.317803 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:57.318212 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:57.318316 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:57.322383 (+ 4067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:57.322388 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:57.322391 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:57.322421 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:57.325134 (+ 2713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:57.325911 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:57.325917 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:57.325920 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:57.326069 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:57.326474 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:57.326479 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:57.326482 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:57.326779 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:57.327077 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:57.327301 (+ 224us) server_negotiation.cc:299] Negotiation successful
0620 18:43:57.327829 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 18:43:57.331926 14572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f68ca29c2dd4016b243df8733326ee9: Bootstrap starting.
I20260620 18:43:57.332933 24012 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:43:57.335397 14572 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f68ca29c2dd4016b243df8733326ee9: Neither blocks nor log segments found. Creating new log.
I20260620 18:43:57.336596 14572 log.cc:824] T 00000000000000000000000000000000 P 2f68ca29c2dd4016b243df8733326ee9: Log is configured to *not* fsync() on all Append() calls
I20260620 18:43:57.335247 14399 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "fcedba0908f545b481a6b7a65cd28ef6"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44866
I20260620 18:43:57.339679 14572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f68ca29c2dd4016b243df8733326ee9: No bootstrap required, opened a new log
I20260620 18:43:57.347848 14572 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f68ca29c2dd4016b243df8733326ee9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcedba0908f545b481a6b7a65cd28ef6" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 34249 } } peers { permanent_uuid: "cabe4f44108242658f56c8cc76a22f31" member_type: VOTER last_known_addr { host: "127.23.115.61" port: 38737 } } peers { permanent_uuid: "2f68ca29c2dd4016b243df8733326ee9" member_type: VOTER last_known_addr { host: "127.23.115.60" port: 33719 } }
I20260620 18:43:57.348377 14572 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f68ca29c2dd4016b243df8733326ee9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f68ca29c2dd4016b243df8733326ee9, State: Initialized, Role: FOLLOWER
I20260620 18:43:57.348868 14572 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f68ca29c2dd4016b243df8733326ee9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcedba0908f545b481a6b7a65cd28ef6" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 34249 } } peers { permanent_uuid: "cabe4f44108242658f56c8cc76a22f31" member_type: VOTER last_known_addr { host: "127.23.115.61" port: 38737 } } peers { permanent_uuid: "2f68ca29c2dd4016b243df8733326ee9" member_type: VOTER last_known_addr { host: "127.23.115.60" port: 33719 } }
I20260620 18:43:57.349738 14578 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f68ca29c2dd4016b243df8733326ee9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcedba0908f545b481a6b7a65cd28ef6" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 34249 } } peers { permanent_uuid: "cabe4f44108242658f56c8cc76a22f31" member_type: VOTER last_known_addr { host: "127.23.115.61" port: 38737 } } peers { permanent_uuid: "2f68ca29c2dd4016b243df8733326ee9" member_type: VOTER last_known_addr { host: "127.23.115.60" port: 33719 } } }
I20260620 18:43:57.350033 14578 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f68ca29c2dd4016b243df8733326ee9 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:43:57.351159 14572 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f68ca29c2dd4016b243df8733326ee9 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 18:43:57.360694 14589 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2f68ca29c2dd4016b243df8733326ee9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 18:43:57.360867 14589 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 18:43:57.359674 14505 master_runner.cc:186] Error getting master registration for 127.23.115.62:34249: OK, instance_id { permanent_uuid: "fcedba0908f545b481a6b7a65cd28ef6" instance_seqno: 1781981036258855 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 18:43:57.371858 14505 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 18:43:57.395825 14424 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.115.60" port: 33719 } attempt: 4
Jun 20 18:43:57 dist-test-slave-d645 krb5kdc[14344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981036, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.62@KRBTEST.COM for kudu/127.23.115.60@KRBTEST.COM
I20260620 18:43:57.410589 14575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:57.396757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33076 (local address 127.23.115.60:33719)
0620 18:43:57.396960 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:57.396965 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:57.396985 (+ 20us) server_negotiation.cc:407] Connection header received
0620 18:43:57.397136 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:57.397141 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:57.397207 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:57.397286 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:57.398055 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:57.398982 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:57.400517 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:57.401309 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:57.401564 (+ 255us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:57.405483 (+ 3919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:57.405489 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:57.405492 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:57.405522 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:57.408055 (+ 2533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:57.408683 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:57.408689 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:57.408692 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:57.408803 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:57.409150 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:57.409155 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:57.409159 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:57.409509 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:57.409633 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:57.410039 (+ 406us) server_negotiation.cc:299] Negotiation successful
0620 18:43:57.410242 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 18:43:57.410584 14426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:57.396637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.60:33719 (local address 127.0.0.1:33076)
0620 18:43:57.396849 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:57.396869 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:57.397002 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:57.397348 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:57.397352 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:57.397758 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:43:57.397918 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:57.397926 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:57.399134 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:57.399140 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:57.400196 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:57.400209 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:57.401569 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:57.401576 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:57.401777 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:57.402444 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:43:57.402469 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:43:57.405294 (+ 2825us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:43:57.408184 (+ 2890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:57.408190 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:57.408194 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:57.408574 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:57.408905 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:57.408910 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:57.408914 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:57.409053 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:57.409635 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:43:57.409640 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:43:57.409793 (+ 153us) client_negotiation.cc:769] Sending connection context
0620 18:43:57.409909 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 18:43:57.410208 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260620 18:43:57.415552 14424 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcedba0908f545b481a6b7a65cd28ef6: Bootstrap starting.
I20260620 18:43:57.418534 14424 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fcedba0908f545b481a6b7a65cd28ef6: Neither blocks nor log segments found. Creating new log.
I20260620 18:43:57.419816 14424 log.cc:824] T 00000000000000000000000000000000 P fcedba0908f545b481a6b7a65cd28ef6: Log is configured to *not* fsync() on all Append() calls
I20260620 18:43:57.422560 14424 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcedba0908f545b481a6b7a65cd28ef6: No bootstrap required, opened a new log
I20260620 18:43:57.427672 14424 raft_consensus.cc:348] T 00000000000000000000000000000000 P fcedba0908f545b481a6b7a65cd28ef6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcedba0908f545b481a6b7a65cd28ef6" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 34249 } } peers { permanent_uuid: "cabe4f44108242658f56c8cc76a22f31" member_type: VOTER last_known_addr { host: "127.23.115.61" port: 38737 } } peers { permanent_uuid: "2f68ca29c2dd4016b243df8733326ee9" member_type: VOTER last_known_addr { host: "127.23.115.60" port: 33719 } }
I20260620 18:43:57.428014 14424 raft_consensus.cc:729] T 00000000000000000000000000000000 P fcedba0908f545b481a6b7a65cd28ef6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcedba0908f545b481a6b7a65cd28ef6, State: Initialized, Role: FOLLOWER
I20260620 18:43:57.428428 14424 consensus_queue.cc:260] T 00000000000000000000000000000000 P fcedba0908f545b481a6b7a65cd28ef6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcedba0908f545b481a6b7a65cd28ef6" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 34249 } } peers { permanent_uuid: "cabe4f44108242658f56c8cc76a22f31" member_type: VOTER last_known_addr { host: "127.23.115.61" port: 38737 } } peers { permanent_uuid: "2f68ca29c2dd4016b243df8733326ee9" member_type: VOTER last_known_addr { host: "127.23.115.60" port: 33719 } }
I20260620 18:43:57.429277 14592 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcedba0908f545b481a6b7a65cd28ef6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcedba0908f545b481a6b7a65cd28ef6" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 34249 } } peers { permanent_uuid: "cabe4f44108242658f56c8cc76a22f31" member_type: VOTER last_known_addr { host: "127.23.115.61" port: 38737 } } peers { permanent_uuid: "2f68ca29c2dd4016b243df8733326ee9" member_type: VOTER last_known_addr { host: "127.23.115.60" port: 33719 } } }
I20260620 18:43:57.429639 14592 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcedba0908f545b481a6b7a65cd28ef6 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:43:57.430348 14424 sys_catalog.cc:564] T 00000000000000000000000000000000 P fcedba0908f545b481a6b7a65cd28ef6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:43:57.430486 14399 raft_consensus.cc:482] T 00000000000000000000000000000000 P fcedba0908f545b481a6b7a65cd28ef6 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 18:43:57.430708 14399 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fcedba0908f545b481a6b7a65cd28ef6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:43:57.434165 14399 raft_consensus.cc:504] T 00000000000000000000000000000000 P fcedba0908f545b481a6b7a65cd28ef6 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcedba0908f545b481a6b7a65cd28ef6" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 34249 } } peers { permanent_uuid: "cabe4f44108242658f56c8cc76a22f31" member_type: VOTER last_known_addr { host: "127.23.115.61" port: 38737 } } peers { permanent_uuid: "2f68ca29c2dd4016b243df8733326ee9" member_type: VOTER last_known_addr { host: "127.23.115.60" port: 33719 } }
I20260620 18:43:57.435838 14399 leader_election.cc:288] T 00000000000000000000000000000000 P fcedba0908f545b481a6b7a65cd28ef6 [CANDIDATE]: Term 1 election: Requested vote from peers cabe4f44108242658f56c8cc76a22f31 (127.23.115.61:38737), 2f68ca29c2dd4016b243df8733326ee9 (127.23.115.60:33719)
I20260620 18:43:57.436794 14603 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 18:43:57.436960 14547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fcedba0908f545b481a6b7a65cd28ef6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "2f68ca29c2dd4016b243df8733326ee9"
I20260620 18:43:57.437427 14547 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f68ca29c2dd4016b243df8733326ee9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:43:57.440729 14547 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2f68ca29c2dd4016b243df8733326ee9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fcedba0908f545b481a6b7a65cd28ef6 in term 1.
I20260620 18:43:57.441303 14374 leader_election.cc:302] T 00000000000000000000000000000000 P fcedba0908f545b481a6b7a65cd28ef6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f68ca29c2dd4016b243df8733326ee9, fcedba0908f545b481a6b7a65cd28ef6; no voters:
I20260620 18:43:57.441742 14592 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fcedba0908f545b481a6b7a65cd28ef6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:43:57.442137 14357 master_runner.cc:186] Error getting master registration for 127.23.115.61:38737: OK, instance_id { permanent_uuid: "cabe4f44108242658f56c8cc76a22f31" instance_seqno: 1781981036750865 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 18:43:57.453408 14357 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 18:43:57.453832 14592 raft_consensus.cc:686] T 00000000000000000000000000000000 P fcedba0908f545b481a6b7a65cd28ef6 [term 1 LEADER]: Becoming Leader. State: Replica: fcedba0908f545b481a6b7a65cd28ef6, State: Running, Role: LEADER
I20260620 18:43:57.454468 14592 consensus_queue.cc:237] T 00000000000000000000000000000000 P fcedba0908f545b481a6b7a65cd28ef6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcedba0908f545b481a6b7a65cd28ef6" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 34249 } } peers { permanent_uuid: "cabe4f44108242658f56c8cc76a22f31" member_type: VOTER last_known_addr { host: "127.23.115.61" port: 38737 } } peers { permanent_uuid: "2f68ca29c2dd4016b243df8733326ee9" member_type: VOTER last_known_addr { host: "127.23.115.60" port: 33719 } }
I20260620 18:43:57.458024 14608 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcedba0908f545b481a6b7a65cd28ef6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fcedba0908f545b481a6b7a65cd28ef6. Latest consensus state: current_term: 1 leader_uuid: "fcedba0908f545b481a6b7a65cd28ef6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcedba0908f545b481a6b7a65cd28ef6" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 34249 } } peers { permanent_uuid: "cabe4f44108242658f56c8cc76a22f31" member_type: VOTER last_known_addr { host: "127.23.115.61" port: 38737 } } peers { permanent_uuid: "2f68ca29c2dd4016b243df8733326ee9" member_type: VOTER last_known_addr { host: "127.23.115.60" port: 33719 } } }
I20260620 18:43:57.458456 14608 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcedba0908f545b481a6b7a65cd28ef6 [sys.catalog]: This master's current role is: LEADER
I20260620 18:43:57.465359 14611 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:43:57.475455 14498 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.115.60" port: 33719 } attempt: 5
I20260620 18:43:57.488588 14611 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260620 18:43:57.490785 14526 tablet.cc:2369] T 00000000000000000000000000000000 P 2f68ca29c2dd4016b243df8733326ee9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:43:57.492154 14502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:57.459179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40802 (local address 127.23.115.61:38737)
0620 18:43:57.459430 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:57.459438 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:57.467574 (+ 8136us) server_negotiation.cc:407] Connection header received
0620 18:43:57.467820 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:57.467827 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:57.467912 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:57.468066 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:57.472698 (+ 4632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:57.473781 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:57.477240 (+ 3459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:57.477940 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:57.478046 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:57.483439 (+ 5393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:57.483446 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:57.483452 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:57.483494 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:57.487418 (+ 3924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:57.489128 (+ 1710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:57.489134 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:57.489138 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:57.489315 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:57.489961 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:57.489967 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:57.489971 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:57.490366 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:57.490624 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:57.491553 (+ 929us) server_negotiation.cc:299] Negotiation successful
0620 18:43:57.491797 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260620 18:43:57.498615 14378 tablet.cc:2369] T 00000000000000000000000000000000 P fcedba0908f545b481a6b7a65cd28ef6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:43:57.502664 14427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:57.452842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44870 (local address 127.23.115.62:34249)
0620 18:43:57.453024 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:57.453030 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:57.461600 (+ 8570us) server_negotiation.cc:407] Connection header received
0620 18:43:57.462019 (+ 419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:57.462024 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:57.462108 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:57.462185 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:57.470339 (+ 8154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:57.471476 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:57.474401 (+ 2925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:57.475158 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:57.475260 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:57.493013 (+ 17753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:57.493021 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:57.493027 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:57.493071 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:57.496591 (+ 3520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:57.500157 (+ 3566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:57.500164 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:57.500167 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:57.500358 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:57.500841 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:57.500846 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:57.500850 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:57.501207 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:57.501470 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:57.502035 (+ 565us) server_negotiation.cc:299] Negotiation successful
0620 18:43:57.502286 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 18:43:57.513375 14575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:57.462078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33084 (local address 127.23.115.60:33719)
0620 18:43:57.462245 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:57.462251 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:57.470243 (+ 7992us) server_negotiation.cc:407] Connection header received
0620 18:43:57.473508 (+ 3265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:57.473516 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:57.473602 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:57.473668 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:57.481824 (+ 8156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:57.483041 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:57.488807 (+ 5766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:57.489530 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:57.489632 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:57.504993 (+ 15361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:57.505000 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:57.505005 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:57.505051 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:57.508651 (+ 3600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:57.510875 (+ 2224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:57.510880 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:57.510883 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:57.511124 (+ 241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:57.511579 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:57.511588 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:57.511591 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:57.511945 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:57.512167 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:57.512732 (+ 565us) server_negotiation.cc:299] Negotiation successful
0620 18:43:57.512966 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64,"spinlock_wait_cycles":24832}
Jun 20 18:43:57 dist-test-slave-d645 krb5kdc[14344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981036, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.115.61@KRBTEST.COM for kudu/127.23.115.60@KRBTEST.COM
I20260620 18:43:57.521837 14547 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2f68ca29c2dd4016b243df8733326ee9 [term 1 FOLLOWER]: Refusing update from remote peer fcedba0908f545b481a6b7a65cd28ef6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:43:57.522785 14608 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fcedba0908f545b481a6b7a65cd28ef6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f68ca29c2dd4016b243df8733326ee9" member_type: VOTER last_known_addr { host: "127.23.115.60" port: 33719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:43:57.524041 14500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:57.476052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.115.60:33719 (local address 127.0.0.1:33086)
0620 18:43:57.476300 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0620 18:43:57.476320 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:43:57.476455 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:43:57.498033 (+ 21578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:43:57.498039 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:43:57.498520 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:43:57.498712 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:57.498723 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:57.508549 (+ 9826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:57.508556 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:57.509936 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:43:57.509953 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:43:57.510886 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:43:57.510892 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:43:57.511142 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:57.511784 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:43:57.511809 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:43:57.514808 (+ 2999us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:43:57.518843 (+ 4035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:57.518849 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:57.518853 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:57.519397 (+ 544us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:57.519827 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:43:57.519832 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:43:57.519835 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:43:57.520006 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:43:57.523029 (+ 3023us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:43:57.523034 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:43:57.523205 (+ 171us) client_negotiation.cc:769] Sending connection context
0620 18:43:57.523354 (+ 149us) client_negotiation.cc:240] Negotiation successful
0620 18:43:57.523634 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141}
I20260620 18:43:57.525002 14615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:57.476107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33086 (local address 127.23.115.60:33719)
0620 18:43:57.497682 (+ 21575us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:57.497688 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:57.497715 (+ 27us) server_negotiation.cc:407] Connection header received
0620 18:43:57.497786 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:57.497791 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:57.497869 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:57.497983 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:57.507035 (+ 9052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:57.508392 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:57.510079 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:57.510724 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:57.510853 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:57.515053 (+ 4200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:43:57.515061 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:43:57.515067 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:43:57.515115 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:43:57.518714 (+ 3599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:57.519502 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:57.519507 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:57.519510 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:57.519661 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:43:57.520119 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:43:57.520139 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:43:57.520142 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:43:57.520504 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:43:57.520635 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:57.524372 (+ 3737us) server_negotiation.cc:299] Negotiation successful
0620 18:43:57.524591 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21422,"thread_start_us":22076,"threads_started":1}
I20260620 18:43:57.532008 14498 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cabe4f44108242658f56c8cc76a22f31: Bootstrap starting.
I20260620 18:43:57.539201 14498 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cabe4f44108242658f56c8cc76a22f31: Neither blocks nor log segments found. Creating new log.
I20260620 18:43:57.540535 14498 log.cc:824] T 00000000000000000000000000000000 P cabe4f44108242658f56c8cc76a22f31: Log is configured to *not* fsync() on all Append() calls
I20260620 18:43:57.546850 14578 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f68ca29c2dd4016b243df8733326ee9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fcedba0908f545b481a6b7a65cd28ef6. Latest consensus state: current_term: 1 leader_uuid: "fcedba0908f545b481a6b7a65cd28ef6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcedba0908f545b481a6b7a65cd28ef6" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 34249 } } peers { permanent_uuid: "cabe4f44108242658f56c8cc76a22f31" member_type: VOTER last_known_addr { host: "127.23.115.61" port: 38737 } } peers { permanent_uuid: "2f68ca29c2dd4016b243df8733326ee9" member_type: VOTER last_known_addr { host: "127.23.115.60" port: 33719 } } }
I20260620 18:43:57.547199 14578 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f68ca29c2dd4016b243df8733326ee9 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:43:57.547968 14498 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cabe4f44108242658f56c8cc76a22f31: No bootstrap required, opened a new log
I20260620 18:43:57.559360 14608 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcedba0908f545b481a6b7a65cd28ef6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fcedba0908f545b481a6b7a65cd28ef6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcedba0908f545b481a6b7a65cd28ef6" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 34249 } } peers { permanent_uuid: "cabe4f44108242658f56c8cc76a22f31" member_type: VOTER last_known_addr { host: "127.23.115.61" port: 38737 } } peers { permanent_uuid: "2f68ca29c2dd4016b243df8733326ee9" member_type: VOTER last_known_addr { host: "127.23.115.60" port: 33719 } } }
I20260620 18:43:57.559687 14608 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcedba0908f545b481a6b7a65cd28ef6 [sys.catalog]: This master's current role is: LEADER
I20260620 18:43:57.559926 14498 raft_consensus.cc:348] T 00000000000000000000000000000000 P cabe4f44108242658f56c8cc76a22f31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcedba0908f545b481a6b7a65cd28ef6" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 34249 } } peers { permanent_uuid: "cabe4f44108242658f56c8cc76a22f31" member_type: VOTER last_known_addr { host: "127.23.115.61" port: 38737 } } peers { permanent_uuid: "2f68ca29c2dd4016b243df8733326ee9" member_type: VOTER last_known_addr { host: "127.23.115.60" port: 33719 } }
I20260620 18:43:57.560349 14498 raft_consensus.cc:729] T 00000000000000000000000000000000 P cabe4f44108242658f56c8cc76a22f31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cabe4f44108242658f56c8cc76a22f31, State: Initialized, Role: FOLLOWER
I20260620 18:43:57.560870 14498 consensus_queue.cc:260] T 00000000000000000000000000000000 P cabe4f44108242658f56c8cc76a22f31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcedba0908f545b481a6b7a65cd28ef6" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 34249 } } peers { permanent_uuid: "cabe4f44108242658f56c8cc76a22f31" member_type: VOTER last_known_addr { host: "127.23.115.61" port: 38737 } } peers { permanent_uuid: "2f68ca29c2dd4016b243df8733326ee9" member_type: VOTER last_known_addr { host: "127.23.115.60" port: 33719 } }
I20260620 18:43:57.562412 14611 catalog_manager.cc:1347] Generated new cluster ID: c296123eaebe40c78c2a485d917fb77d
I20260620 18:43:57.562507 14611 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:43:57.562369 14578 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f68ca29c2dd4016b243df8733326ee9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fcedba0908f545b481a6b7a65cd28ef6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcedba0908f545b481a6b7a65cd28ef6" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 34249 } } peers { permanent_uuid: "cabe4f44108242658f56c8cc76a22f31" member_type: VOTER last_known_addr { host: "127.23.115.61" port: 38737 } } peers { permanent_uuid: "2f68ca29c2dd4016b243df8733326ee9" member_type: VOTER last_known_addr { host: "127.23.115.60" port: 33719 } } }
I20260620 18:43:57.562599 14578 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f68ca29c2dd4016b243df8733326ee9 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:43:57.575038 14622 sys_catalog.cc:455] T 00000000000000000000000000000000 P cabe4f44108242658f56c8cc76a22f31 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcedba0908f545b481a6b7a65cd28ef6" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 34249 } } peers { permanent_uuid: "cabe4f44108242658f56c8cc76a22f31" member_type: VOTER last_known_addr { host: "127.23.115.61" port: 38737 } } peers { permanent_uuid: "2f68ca29c2dd4016b243df8733326ee9" member_type: VOTER last_known_addr { host: "127.23.115.60" port: 33719 } } }
I20260620 18:43:57.577410 14622 sys_catalog.cc:458] T 00000000000000000000000000000000 P cabe4f44108242658f56c8cc76a22f31 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:43:57.577310 14473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fcedba0908f545b481a6b7a65cd28ef6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "cabe4f44108242658f56c8cc76a22f31"
I20260620 18:43:57.577263 14498 sys_catalog.cc:564] T 00000000000000000000000000000000 P cabe4f44108242658f56c8cc76a22f31 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:43:57.577826 14473 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cabe4f44108242658f56c8cc76a22f31 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:43:57.580677 14611 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:43:57.581543 14473 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cabe4f44108242658f56c8cc76a22f31 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fcedba0908f545b481a6b7a65cd28ef6 in term 1.
I20260620 18:43:57.582448 14472 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cabe4f44108242658f56c8cc76a22f31 [term 1 FOLLOWER]: Refusing update from remote peer fcedba0908f545b481a6b7a65cd28ef6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:43:57.582674 14611 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:43:57.583379 14608 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fcedba0908f545b481a6b7a65cd28ef6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cabe4f44108242658f56c8cc76a22f31" member_type: VOTER last_known_addr { host: "127.23.115.61" port: 38737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:43:57.604471 14611 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fcedba0908f545b481a6b7a65cd28ef6: Generated new TSK 0
I20260620 18:43:57.605593 14611 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:43:57.626114 14622 sys_catalog.cc:455] T 00000000000000000000000000000000 P cabe4f44108242658f56c8cc76a22f31 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fcedba0908f545b481a6b7a65cd28ef6. Latest consensus state: current_term: 1 leader_uuid: "fcedba0908f545b481a6b7a65cd28ef6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcedba0908f545b481a6b7a65cd28ef6" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 34249 } } peers { permanent_uuid: "cabe4f44108242658f56c8cc76a22f31" member_type: VOTER last_known_addr { host: "127.23.115.61" port: 38737 } } peers { permanent_uuid: "2f68ca29c2dd4016b243df8733326ee9" member_type: VOTER last_known_addr { host: "127.23.115.60" port: 33719 } } }
I20260620 18:43:57.626467 14622 sys_catalog.cc:458] T 00000000000000000000000000000000 P cabe4f44108242658f56c8cc76a22f31 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:43:57.626946 14631 sys_catalog.cc:455] T 00000000000000000000000000000000 P cabe4f44108242658f56c8cc76a22f31 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fcedba0908f545b481a6b7a65cd28ef6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcedba0908f545b481a6b7a65cd28ef6" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 34249 } } peers { permanent_uuid: "cabe4f44108242658f56c8cc76a22f31" member_type: VOTER last_known_addr { host: "127.23.115.61" port: 38737 } } peers { permanent_uuid: "2f68ca29c2dd4016b243df8733326ee9" member_type: VOTER last_known_addr { host: "127.23.115.60" port: 33719 } } }
I20260620 18:43:57.627900 14631 sys_catalog.cc:458] T 00000000000000000000000000000000 P cabe4f44108242658f56c8cc76a22f31 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:43:57.628504 14592 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcedba0908f545b481a6b7a65cd28ef6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fcedba0908f545b481a6b7a65cd28ef6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcedba0908f545b481a6b7a65cd28ef6" member_type: VOTER last_known_addr { host: "127.23.115.62" port: 34249 } } peers { permanent_uuid: "cabe4f44108242658f56c8cc76a22f31" member_type: VOTER last_known_addr { host: "127.23.115.61" port: 38737 } } peers { permanent_uuid: "2f68ca29c2dd4016b243df8733326ee9" member_type: VOTER last_known_addr { host: "127.23.115.60" port: 33719 } } }
I20260620 18:43:57.628837 14592 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcedba0908f545b481a6b7a65cd28ef6 [sys.catalog]: This master's current role is: LEADER
I20260620 18:43:57.651491 14639 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260620 18:43:57.697583 14502 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:43:57.683977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51942 (local address 127.23.115.61:38737)
0620 18:43:57.684187 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:57.684194 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:57.695542 (+ 11348us) server_negotiation.cc:407] Connection header received
0620 18:43:57.695869 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:57.695875 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:57.695977 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:57.696051 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:57.697327 (+ 1276us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51942: BlockingRecv error: recv got EOF from 127.0.0.1:51942 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260620 18:43:57.699716 14427 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:43:57.683848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36050 (local address 127.23.115.62:34249)
0620 18:43:57.684088 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:57.684096 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:57.698178 (+ 14082us) server_negotiation.cc:407] Connection header received
0620 18:43:57.698409 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:57.698416 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:57.698499 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:57.698566 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:57.699486 (+ 920us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36050: BlockingRecv error: recv got EOF from 127.0.0.1:36050 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
W20260620 18:43:57.701684 14615 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:43:57.695868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34814 (local address 127.23.115.60:33719)
0620 18:43:57.696045 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:57.696050 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:57.700212 (+ 4162us) server_negotiation.cc:407] Connection header received
0620 18:43:57.700434 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:57.700440 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:57.700541 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:57.700610 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:57.701461 (+ 851us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34814: BlockingRecv error: recv got EOF from 127.0.0.1:34814 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260620 18:43:57.715157 14502 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:43:57.713027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51956 (local address 127.23.115.61:38737)
0620 18:43:57.713243 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:57.713250 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:57.713665 (+ 415us) server_negotiation.cc:407] Connection header received
0620 18:43:57.713880 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:57.713884 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:57.713983 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:57.714068 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:57.714870 (+ 802us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51956: BlockingRecv error: recv got EOF from 127.0.0.1:51956 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
W20260620 18:43:57.731244 14502 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:43:57.729255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51964 (local address 127.23.115.61:38737)
0620 18:43:57.729456 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:57.729464 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:57.729954 (+ 490us) server_negotiation.cc:407] Connection header received
0620 18:43:57.730174 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:57.730178 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:57.730248 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:57.730331 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:57.731049 (+ 718us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51964: BlockingRecv error: recv got EOF from 127.0.0.1:51964 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260620 18:43:57.738606 14502 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:43:57.736719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51966 (local address 127.23.115.61:38737)
0620 18:43:57.736913 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:57.736920 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:57.737374 (+ 454us) server_negotiation.cc:407] Connection header received
0620 18:43:57.737552 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:57.737556 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:57.737634 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:57.737716 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:57.738380 (+ 664us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51966: BlockingRecv error: recv got EOF from 127.0.0.1:51966 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260620 18:43:57.752774 14502 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:43:57.748592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51968 (local address 127.23.115.61:38737)
0620 18:43:57.748837 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:57.748843 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:57.751477 (+ 2634us) server_negotiation.cc:407] Connection header received
0620 18:43:57.751660 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:57.751664 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:57.751735 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:57.751819 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:57.752593 (+ 774us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51968: BlockingRecv error: recv got EOF from 127.0.0.1:51968 (error 108)
Metrics: {"server-negotiator.queue_time_us":110}
W20260620 18:43:57.787225 14502 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:43:57.785131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51984 (local address 127.23.115.61:38737)
0620 18:43:57.785350 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:57.785357 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:57.785877 (+ 520us) server_negotiation.cc:407] Connection header received
0620 18:43:57.786066 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:57.786071 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:57.786143 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:57.786225 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:57.787003 (+ 778us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51984: BlockingRecv error: recv got EOF from 127.0.0.1:51984 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260620 18:43:57.808312 14502 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:43:57.806318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51990 (local address 127.23.115.61:38737)
0620 18:43:57.806537 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:57.806544 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:57.806999 (+ 455us) server_negotiation.cc:407] Connection header received
0620 18:43:57.807240 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:57.807244 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:57.807330 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:57.807416 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:57.808084 (+ 668us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51990: BlockingRecv error: recv got EOF from 127.0.0.1:51990 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
W20260620 18:43:57.846909 14502 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:43:57.844722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52002 (local address 127.23.115.61:38737)
0620 18:43:57.844933 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:57.844941 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:57.845418 (+ 477us) server_negotiation.cc:407] Connection header received
0620 18:43:57.845651 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:57.845657 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:57.845794 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:57.845883 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:57.846645 (+ 762us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52002: BlockingRecv error: recv got EOF from 127.0.0.1:52002 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260620 18:43:57.916431 14502 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:43:57.914200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52016 (local address 127.23.115.61:38737)
0620 18:43:57.914437 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:57.914444 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:57.914920 (+ 476us) server_negotiation.cc:407] Connection header received
0620 18:43:57.915211 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:57.915216 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:57.915294 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:57.915380 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:57.916196 (+ 816us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52016: BlockingRecv error: recv got EOF from 127.0.0.1:52016 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260620 18:43:58.049611 14502 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:43:58.047814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52024 (local address 127.23.115.61:38737)
0620 18:43:58.047972 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:58.047978 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:58.048330 (+ 352us) server_negotiation.cc:407] Connection header received
0620 18:43:58.048487 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:58.048491 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:58.048570 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:58.048716 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:58.049426 (+ 710us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52024: BlockingRecv error: recv got EOF from 127.0.0.1:52024 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260620 18:43:58.310899 14502 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:43:58.309293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52040 (local address 127.23.115.61:38737)
0620 18:43:58.309450 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:58.309454 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:58.309666 (+ 212us) server_negotiation.cc:407] Connection header received
0620 18:43:58.309818 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:58.309823 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:58.309899 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:58.309992 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:43:58.310709 (+ 717us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52040: BlockingRecv error: recv got EOF from 127.0.0.1:52040 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 18:43:58.362908 14589 catalog_manager.cc:1259] Loaded cluster ID: c296123eaebe40c78c2a485d917fb77d
I20260620 18:43:58.363042 14589 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2f68ca29c2dd4016b243df8733326ee9: loading cluster ID for follower catalog manager: success
I20260620 18:43:58.365476 14589 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2f68ca29c2dd4016b243df8733326ee9: acquiring CA information for follower catalog manager: success
I20260620 18:43:58.367249 14589 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2f68ca29c2dd4016b243df8733326ee9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 18:43:58.656569 14639 catalog_manager.cc:1259] Loaded cluster ID: c296123eaebe40c78c2a485d917fb77d
I20260620 18:43:58.656706 14639 catalog_manager.cc:1552] T 00000000000000000000000000000000 P cabe4f44108242658f56c8cc76a22f31: loading cluster ID for follower catalog manager: success
I20260620 18:43:58.659541 14639 catalog_manager.cc:1574] T 00000000000000000000000000000000 P cabe4f44108242658f56c8cc76a22f31: acquiring CA information for follower catalog manager: success
I20260620 18:43:58.661458 14639 catalog_manager.cc:1602] T 00000000000000000000000000000000 P cabe4f44108242658f56c8cc76a22f31: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 18:43:58.839272 14705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:58.832407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52054 (local address 127.23.115.61:38737)
0620 18:43:58.833275 (+ 868us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:58.833282 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:58.833305 (+ 23us) server_negotiation.cc:407] Connection header received
0620 18:43:58.833386 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:58.833390 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:58.833525 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:58.833640 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:43:58.834161 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:58.835003 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:58.836832 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:58.837293 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:58.837395 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:58.837986 (+ 591us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:43:58.838513 (+ 527us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:43:58.838636 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:58.838776 (+ 140us) server_negotiation.cc:299] Negotiation successful
0620 18:43:58.838850 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":766,"thread_start_us":595,"threads_started":1}
I20260620 18:43:58.850972 14711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:58.845227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34816 (local address 127.23.115.60:33719)
0620 18:43:58.846047 (+ 820us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:58.846051 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:58.846065 (+ 14us) server_negotiation.cc:407] Connection header received
0620 18:43:58.846150 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:58.846153 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:58.846288 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:58.846451 (+ 163us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:43:58.846830 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:58.847611 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:58.848857 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:58.849241 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:58.849353 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:58.849825 (+ 472us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:43:58.850289 (+ 464us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:43:58.850437 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:58.850547 (+ 110us) server_negotiation.cc:299] Negotiation successful
0620 18:43:58.850630 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":729,"thread_start_us":634,"threads_started":1}
I20260620 18:43:58.871379 14705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:58.856989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52060 (local address 127.23.115.61:38737)
0620 18:43:58.857175 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:58.857182 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:58.864457 (+ 7275us) server_negotiation.cc:407] Connection header received
0620 18:43:58.864544 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:58.864550 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:58.864633 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:58.864721 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:43:58.865325 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:58.866595 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:58.869046 (+ 2451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:58.869719 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:58.869815 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:58.870478 (+ 663us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:43:58.870623 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:43:58.870686 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:58.870909 (+ 223us) server_negotiation.cc:299] Negotiation successful
0620 18:43:58.871025 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 18:43:58.872792 14718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:58.856608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36054 (local address 127.23.115.62:34249)
0620 18:43:58.864503 (+ 7895us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:58.864510 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:58.864529 (+ 19us) server_negotiation.cc:407] Connection header received
0620 18:43:58.864642 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:58.864648 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:58.864864 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:58.865668 (+ 804us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:43:58.865831 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:58.866874 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:58.869067 (+ 2193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:58.869668 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:58.870277 (+ 609us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:58.870468 (+ 191us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:43:58.870874 (+ 406us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:43:58.872380 (+ 1506us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:58.872432 (+ 52us) server_negotiation.cc:299] Negotiation successful
0620 18:43:58.872517 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7769,"thread_start_us":7659,"threads_started":1}
I20260620 18:43:58.877090 14711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:43:58.864358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34830 (local address 127.23.115.60:33719)
0620 18:43:58.864510 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 18:43:58.864515 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:43:58.871939 (+ 7424us) server_negotiation.cc:407] Connection header received
0620 18:43:58.872015 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:43:58.872020 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:43:58.872120 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:43:58.872211 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:43:58.872639 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:58.873590 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:58.875302 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:43:58.875894 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:43:58.876014 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:43:58.876404 (+ 390us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:43:58.876517 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:43:58.876589 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 18:43:58.876726 (+ 137us) server_negotiation.cc:299] Negotiation successful
0620 18:43:58.876803 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 18:43:58.877238 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 14357
I20260620 18:43:58.878240 14419 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:43:59.005380 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 14357
I20260620 18:43:59.025367 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 14431
I20260620 18:43:59.026530 14493 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:43:59.130342 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 14431
I20260620 18:43:59.147435 24012 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6x5P9o/build/asan/bin/kudu pid 14505
I20260620 18:43:59.148537 14567 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:43:59.252861 24012 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6x5P9o/build/asan/bin/kudu with pid 14505
2026-06-20T18:43:59Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4704 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9165 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (362963 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST